[rrd-developers] rrdupates per second

[Tobias Oetiker]

Hi Bernard,

> I am one of the developers of Ganglia http://www.ganglia.info.  The
> issue we have is that once Ganglia starts monitoring past a certain
> number of hosts (> 1k nodes), we noticed the system grinds to a halt
> with a lot of I/O requests (from rrdupdates).

the prime method to gain performance is to have more cache
available, since this will enable the system to be able to keep all
hot rrd blocks in memory and thus only 'modify-write' disk blocks
as opposed 'read-modify-write' them. (It will have todo reads too,
but only ocasionally).

Since 1.2.24 or so, rrdtool tries to inform the OS (tested on
linux) that it should not read-a-head when accessing rrdfiles, but
just keep the current block around. By this we use less cache per
rrdfile and thus can keep more hot-blocks in memory.

If you do not see this improvement then maybe your OS does not
support the fadvise call or rather configure did not pick it up
correctly.

In 1.3 the whole fadvise/madvise thing has been further tuned, and
on top of that all the IO is mmaped which cal also help.

> Typically Ganglia monitors 32 unique metrics for each host and these
> translate to individual rrd files.  So with 1k nodes, you essentially
> have over 30,000 rrd files that need to be updated periodically.
>
> The current "workaround" for this issue, is to place the rrd files in
> tmpfs.  It would be nice to have a better solution than this.
>
> I did some tests with RRDtool 1.3.  While it does provide some I/O
> enhancements over 1.2.x, the system with standard 7200RPM SATA HD is
> still bogged down by I/O and becomes really sluggish.
>
> I was wondering if you have any suggestions on how to programmatically
> tune the performance?  The Ganglia code responsible for doing the
> rrdupate is written in C.  Our frontend, written in PHP does call
> rrdgraph via system call, but that is not really the bottleneck for
> us.

does this mean ganglia calls rrd_update and links librrd ? or does
it call out?

cheers
tobi


-- 
Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten
http://it.oetiker.ch tobi at oetiker.ch ++41 62 213 9902