[rrd-developers] rrdcached performance problem

[Thorsten von Eicken]

Florian Forster wrote:
>> 547,131,570  < rrd_daemon.c:queue_thread_main'2 (263x) [/usr/bin/rrdcached]
>>   3,186,524  < rrd_daemon.c:queue_thread_main (3x) [/usr/bin/rrdcached]
>> 550,318,094  *  rrd_update.c:rrd_update_r [/usr/lib64/librrd_th.so.4.1.0]
>>     
>
> I wonder where this second update thread is coming from. Did you
> configure one or two? If you configured two, why isn't the second one
> doing any real work?
>   
Yes, there are two configured. Your question is a good one. If I get to 
run callgrind again I'll run it a bit longer and we'll see whether it 
was due to the short observation window. I don't know how the 
locking/dequeueing primitives work, is there perhaps something that 
prefers one thread and the second doesn't run if the first one is fast 
enough at dequeueing?

> spends all of its time (more than 99 %) in “realloc”. So in consequence
> 37.1 % of the time it spent in “handle_request_update” the daemon is
> actually waiting for “realloc”. This is (to me) very unexpected and a
> schoolbook example of “measure before you optimize”.
>
> I think we can get rid of this bottleneck by writing a specialized
> version of “rrd_add_strdup” which reallocates powers of ten. Something
> like:
>
> [...]
>
> It'd be great if you could give the attached patch a try
Test is running, including Kevin's simplification... Thanks for the help!!!

By the way, we were calculating some stats the other day and noticed 
that we're collecting 680k RRDs (most have 1 DS) every 20 seconds. 
That's 35K RRD updates per second. No, this is not on one server...

Cheers,
Thorsten