[rrd-developers] rrdcached performance problem

[kevin brintnall]

On Fri, Oct 30, 2009 at 09:19:30AM -0700, Thorsten von Eicken wrote:
> Thorsten von Eicken wrote:
> >> 37.1 % of the time it spent in ?handle_request_update? the daemon is
> >> actually waiting for ?realloc?. This is (to me) very unexpected and a
> >> schoolbook example of ?measure before you optimize?.
> >>
> >> I think we can get rid of this bottleneck by writing a specialized
> >> version of ?rrd_add_strdup? which reallocates powers of ten. Something
> >> like:
> >>
> >> [...]
> >>
> >> It'd be great if you could give the attached patch a try
> > spends all of its time (more than 99 %) in ?realloc?. So in consequence
> > Test is running, including Kevin's simplification... Thanks for the 
> > help!!!
> Things are again looking much better, almost great I should say! The one 
> thing that still makes me a bit uncomfortable is that at the end of the 
> second hour of run-time there was a cpu spike which caused collectd to 
> grow rapidly. (Still using -w 3600 -z 3600 -f 7200, I put a load of ~50k 
> tree nodes right from the start.) You can see the graphs at 
> http://www.voneicken.com/dl/rrd/ look for the rrdcached-6* series. It 
> flattens out nicely after the spike, but it's one of those things that 
> tend to bite sooner or later. I'm not sure what to do about it.

Thorsten,

What's the start time on that daemon?  If it was started slightly before
the graphs (~21:30-21:35), then it's possible that the CPU increase is
associated with the first flush (-f).

Something is queueing a lot of files; I would only expect that to result
from the flush process if writing had stopped for a set of your RRDs.  For
example, if you have a large number of RRD files in the journal that
aren't being re-written afterwards, then the first flush would contain all
those.

Did you use journal on startup for this test?

If you could profile during that "hump" it would be very instructive...

-- 
 kevin brintnall =~ /kbrint at rufus.net/