[rrd-users] Disk I/O

[Dave Plonka]

On Thu, Mar 13, 2008 at 04:55:37PM -0500, Jeremy wrote:
<snip>
> The RRD files on this server live on the same partition as the OS as well as
> the Nagios software we're running so its a little hard to isolate. However
> if I disable PNP so no RRD files get updated anymore the "util %" as
> reported by iostat drops off to near 0 in a hurry and does not spike back up
> to 100% for more than a split second only occasionally. With PNP disabled
> here's the "sar -d 1 10" average:
> 
> Average:       DEV              tps    rd_sec/s  wr_sec/s
> Average:       dev8-0          77.50      0.00   3587.20
> 
> ...and using "iostat -d -x 1" the %util peaks up to 50-60% occasionally but
> stays near 0 most of the time.

OK, well, that rd_sec/s of zero is a good indication that your buffer
cache is in good shape for the RRD files.  It means that even though
the rrdupdates read blocks from the files (this is necessary to do
the updates), they're being gotten from cache - no disk read necessary.

So the "bad" news is that the newer rrdtool solves a different problem
than what you're observing.

Presumably then your wr_sec/s is just due to the amount and frequency
of RRD files you have.

Anything above and beyond that disk I/O is stuff "unrelated" to RRD.
(nagios or whatever... personally, I know nothing about that... but
that's the place to go next.)
 
<snip>
> Well I'm only anxious because its not my money exactly ;-) I work for a
> fairly hosting company so setting up another server just for graphing would
> be no problem, if it really comes to that, but you have give me renewed
> hope, hehe.

So far we've determined you're not memory limited, you're caching RRD
files well, and there's just a bunch of non-RRD disk I/O that may be
causing you to approach the write capacity of your (single?) disk.

One last suggestion I have is to split the I/O type or software
subsystems to separate file systems on seperate disks, and in a fashion
(controller-wise) that improves your total I/O capacity (across those
multiple disks.)  If RRD represents 1/3 to 1/2 of your I/O that's a
good candidate to be seperated from the other stuff, and its a win
also in that the I/O stats will be on seperate devices, so you'll have
better information in the future.

Dave

-- 
plonka at doit.wisc.edu  http://net.doit.wisc.edu/~plonka/  Madison, WI