[rrd-developers] reducing disk IO load (wish list)

[Dave Plonka]

On Wed, May 30, 2007 at 03:41:48PM +1000, Ian Holsman wrote:
> Kevin Stewart wrote:
> > I am using cricket to collect data for 900 or so devices
> > and have run into some IO problems that seem to stem from the format of
> > the RRDs, correct me if I am wrong, a header block and a bunch of
> > concatenated arrays.
> >   
> Tobi Commented on the file structure, but as for reducing the IO load, 

Yes, the ds values are "interleaved" already as you say.  The part that
is not interleaved is multiple RRAs that have the same pdp_per_row
values (such as both an AVERAGE and a MAX RRA with pdp count of 1,
as with MRTG).

> have you looked at the recent fadvise() patch which was posted to the 
> list in the last week or two?

On this point, Tobi wrote that it will be in rrdtool-1.2.24.

Yes, there's not really a point to trying other strategies unless
you've already done this easy one.  Conservatively, on Linux 2.6.5 or
greater, it approx. triples performance/capacity of systems that have
MRTG-like RRD files (i.e. 2 DSes, 8 RRAs with like pdp_per_row vals
for AVERAGE and MAX).  This performance can vary a bit since the hot
pages and resulting page faults or page replacements are determined
by the RRD file def'n.

If you're anxious, you can find the fadvise RANDOM patch here:

   http://net.doit.wisc.edu/~plonka/rrdtool_fadvise/

while I wrote it for 1.0.49, it can be hand-applied by adding the
call to posix_fadvise as shown in the patch just after the fopen.

Dave

-- 
plonka at doit.wisc.edu  http://net.doit.wisc.edu/~plonka/  Madison, WI