[rrd-users] Disk I/O

[Jeremy]

> As an experiment, I'd cat all your RRD files to /dev/null, then run
> fincore to see what percentage of them are in buffer cache, and also
> see how much free mem you have (with top).  If you have >4GB of RRD
> files, I'd expect there to be near zero free memory afterward, on a
> properly running Linux system.

I will try out fincore tomorrow before and after cat'ing everything to
/dev/null and report the results, thanks for the tip!

> I don't understand the measure of disk I/O you are using...
> Now you said 100% and before you said 500 writes per second
> or some low number.   To compare apples-to-apples, I suggest
> "sar -d [1 10]" and grep for the specific device file on
> which the file system that holds just the RRD files resides,
> so that you see the RRD I/O in isolation from unrelated disk
> I/O.  (sar shows reads and writes per second.)
> Perhaps your high I/O has nothing to do with RRD.

I was using "iostat -d -x 1" to watch the IO info. The ~400 writes/second I
was quoting was write requests per second, the actual number of sectors
written to per sec (which seems to be what sar shows) was much higher. Doing
a "sar -d 1 10" the average it came up with was this:

Average:       DEV              tps    rd_sec/s  wr_sec/s
Average:       dev8-0         501.25      7.99   9685.86

That matches up to what "iostat -d -x 1" whos awas the wsec/s stat (sectors
written per sec)

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s
avgrq-sz avgqu-sz   await  svctm  %util
sda          0.00 347.06  1.96 507.84    7.84 6917.65     3.92  3458.82
13.58   107.24  155.38   1.93  98.14

The RRD files on this server live on the same partition as the OS as well as
the Nagios software we're running so its a little hard to isolate. However
if I disable PNP so no RRD files get updated anymore the "util %" as
reported by iostat drops off to near 0 in a hurry and does not spike back up
to 100% for more than a split second only occasionally. With PNP disabled
here's the "sar -d 1 10" average:

Average:       DEV              tps    rd_sec/s  wr_sec/s
Average:       dev8-0          77.50      0.00   3587.20

...and using "iostat -d -x 1" the %util peaks up to 50-60% occasionally but
stays near 0 most of the time.

I've heard about a new "iotop" tool that shows i/o stats on a per-process
basis but unfortunately the kernel on this box is not new enough to support
that.

> You seem anxious to spend money.  If you measure things to find
> out the source of the performance problem, you can point your
> money in the right direction - or perhaps find out that it's
> not necessary. :-)

Well I'm only anxious because its not my money exactly ;-) I work for a
fairly hosting company so setting up another server just for graphing would
be no problem, if it really comes to that, but you have give me renewed
hope, hehe.

Thanks again for all the info,
Jeremy
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.oetiker.ch/pipermail/rrd-users/attachments/20080313/9ac74435/attachment-0001.html