[rrd-developers] rrdcached performance problem
Thorsten von Eicken
tve at voneicken.com
Fri Oct 30 05:55:07 CET 2009
Florian Forster wrote:
>> 547,131,570 < rrd_daemon.c:queue_thread_main'2 (263x) [/usr/bin/rrdcached]
>> 3,186,524 < rrd_daemon.c:queue_thread_main (3x) [/usr/bin/rrdcached]
>> 550,318,094 * rrd_update.c:rrd_update_r [/usr/lib64/librrd_th.so.4.1.0]
> I wonder where this second update thread is coming from. Did you
> configure one or two? If you configured two, why isn't the second one
> doing any real work?
Yes, there are two configured. Your question is a good one. If I get to
run callgrind again I'll run it a bit longer and we'll see whether it
was due to the short observation window. I don't know how the
locking/dequeueing primitives work, is there perhaps something that
prefers one thread and the second doesn't run if the first one is fast
enough at dequeueing?
> spends all of its time (more than 99 %) in “realloc”. So in consequence
> 37.1 % of the time it spent in “handle_request_update” the daemon is
> actually waiting for “realloc”. This is (to me) very unexpected and a
> schoolbook example of “measure before you optimize”.
> I think we can get rid of this bottleneck by writing a specialized
> version of “rrd_add_strdup” which reallocates powers of ten. Something
> It'd be great if you could give the attached patch a try
Test is running, including Kevin's simplification... Thanks for the help!!!
By the way, we were calculating some stats the other day and noticed
that we're collecting 680k RRDs (most have 1 DS) every 20 seconds.
That's 35K RRD updates per second. No, this is not on one server...
More information about the rrd-developers