[rrd-developers] rrdcached contention when flushing

[Daniel.Pocock at barclayscapital.com]

I've got about 50,000 RRDs updated every 10 seconds (each with 6 RRAs)

When I refresh a ganglia page, it invokes about 20 different rrdtool
graph processes simultaneously.  Each process connects to the rrdcached
daemon over the socket and sends the flush command.

I'm noticing a lot of threads get stuck like this:

(gdb) thread 4
[Switching to thread 4 (Thread 1524803904 (LWP 16876))]#0
0x000000331340a4a6 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
(gdb) bt
#0  0x000000331340a4a6 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1  0x0000000000403bdb in flush_file (
    filename=0x10382ed6
"/.../rrds/cluster/server/net_received_eth1.rrd") at rrd_daemon.c:1048
#2  0x000000000040409d in handle_request_flush (sock=0x2abcfc00d6d0,
buffer=0x10382f39 "d", buffer_size=0)
    at rrd_daemon.c:1261
#3  0x0000000000404f4c in handle_request (sock=0x2abcfc00d6d0,
now=1225732323, buffer=0x10382ed0 "flush", 
    buffer_size=99) at rrd_daemon.c:1640
#4  0x00000000004059fb in connection_thread_main (args=0x2abcfc00d6d0)
at rrd_daemon.c:1987
#5  0x00000033134062f7 in start_thread () from /lib64/libpthread.so.0
#6  0x00000033120ce85d in clone () from /lib64/libc.so.6


They all become un-stuck at the same time, maybe 20 seconds later, and
then the graphs appear very quickly.

I'm using r1621 + the patch adding pthread_cond_init(&ci->flushed,
NULL);

_______________________________________________

This e-mail may contain information that is confidential, privileged or otherwise protected from disclosure. If you are not an intended recipient of this e-mail, do not duplicate or redistribute it by any means. Please delete it and any attachments and notify the sender that you have received it in error. Unless specifically indicated, this e-mail is not an offer to buy or sell or a solicitation to buy or sell any securities, investment products or other financial product or service, an official confirmation of any transaction, or an official statement of Barclays. Any views or opinions presented are solely those of the author and do not necessarily represent those of Barclays. This e-mail is subject to terms available at the following link: www.barcap.com/emaildisclaimer. By messaging with Barclays you consent to the foregoing.  Barclays Capital is the investment banking division of Barclays Bank PLC, a company registered in England (number 1026167) with its registered office at 1 Churchill Place, London, E14 5HP.  This email may relate to or be sent from other members of the Barclays Group.
_______________________________________________