[rrd-users] rrdcached issues with larger number of clients via network/pthread

[Steve Shipway]

We also had this problem, using the trunk version of 1.4.4, but I thought it was due to a separate memory leak issue (now fixed in trunk).

If pthread_create fails then you're out of resources, possibly memory or threads... check your system thread/process limits are not causing issues, and that you have sufficient memory.  Keep an eye on how much memory rrdcached is using and see if it starts getting unfeasibly larger than it is after the first hour.  I've also found that the MRTG daemon (which is writing to rrdcached) can become confused and require restarting at this point, too.

Steve

Steve Shipway
University of Auckland ITS
UNIX Systems Design Lead
s.shipway at auckland.ac.nz
Ph: +64 9 373 7599 ext 86487


________________________________________
From: rrd-users-bounces+s.shipway=auckland.ac.nz at lists.oetiker.ch [rrd-users-bounces+s.shipway=auckland.ac.nz at lists.oetiker.ch] on behalf of Ulf Zimmermann [ulf at openlane.com]
Sent: Sunday, 31 October 2010 6:33 p.m.
To: 'rrd-users at lists.oetiker.ch'
Subject: [rrd-users] rrdcached issues with larger number of clients via network/pthread

I got close to 300 machines running collectd, configured to use unixsocks to rrdcached on a central server. We are running more and more into threads dieing (collectd then starts complaining and fills up /var/messages) and when we try to restart collectd, sometimes it works, sometimes we end up with:

Oct 30 22:27:19 log02 rrdcached[16864]: listen_thread_main: pthread_create failed.
Oct 30 22:27:34 log02 rrdcached[16864]: listen_thread_main: pthread_create failed.
Oct 30 22:28:10 log02 rrdcached[16864]: listen_thread_main: pthread_create failed.

And at this point we usual have to restart the rrdcached daemon, which then means having to restart collectd on close to 300 machines.

How can this be debugged to find the issue (potential inside of pthreads). The central server is running RedHat EL5 Update 4, the rrdtool/rrdcached is 1.4.4 from rpmforge.

Ulf, who is getting more grey hair by the minute with issues like this :-(

_______________________________________________
rrd-users mailing list
rrd-users at lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users