[rrd-users] rrdcached issues with larger number of clients via network/pthread

[Ulf Zimmermann]

If I am running into memory, I am close to the top even shortly after restart:

29837 collectd  15   0 2990m  55m  772 S  8.0  0.3   1:44.38 rrdcached                                                       

This is a 32-bit installation right now (I was going to go 64-bit, but had issue with .. rrdtool, although now with rrdcached I could get around that).

288 connected machines right now:

log02 ulf /home/ulf $ netstat -an | grep 42217 | grep ESTA | wc -l
288


> -----Original Message-----
> From: Tobias Oetiker [mailto:tobi at oetiker.ch]
> Sent: Sunday, October 31, 2010 12:10 AM
> To: Ulf Zimmermann
> Cc: 'rrd-users at lists.oetiker.ch'
> Subject: Re: [rrd-users] rrdcached issues with larger number of clients
> via network/pthread
> 
> HI Ulf,
> 
> Yesterday Ulf Zimmermann wrote:
> 
> > I got close to 300 machines running collectd, configured to use
> unixsocks to rrdcached on a central server. We are running more and
> more into threads dieing (collectd then starts complaining and fills up
> /var/messages) and when we try to restart collectd, sometimes it works,
> sometimes we end up with:
> >
> > Oct 30 22:27:19 log02 rrdcached[16864]: listen_thread_main:
> pthread_create failed.
> > Oct 30 22:27:34 log02 rrdcached[16864]: listen_thread_main:
> pthread_create failed.
> > Oct 30 22:28:10 log02 rrdcached[16864]: listen_thread_main:
> pthread_create failed.
> >
> > And at this point we usual have to restart the rrdcached daemon,
> which then means having to restart collectd on close to 300 machines.
> >
> > How can this be debugged to find the issue (potential inside of
> pthreads). The central server is running RedHat EL5 Update 4, the
> rrdtool/rrdcached is 1.4.4 from rpmforge.
> >
> > Ulf, who is getting more grey hair by the minute with issues like
> this :-(
> 
> try the latest stable snapshot ... there are already a number of
> fixes in the 1.4 branche ... for memory issues and such ... maybe
> this affects your problem too.
> 
> http://oss.oetiker.ch/rrdtool/pub/beta/rrdtool-1.4-svn-snap.tar.gz
> 
> cheers
> tobi
> 
> >
> > _______________________________________________
> > rrd-users mailing list
> > rrd-users at lists.oetiker.ch
> > https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users
> >
> >
> 
> --
> Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland
> http://it.oetiker.ch tobi at oetiker.ch ++41 62 775 9902 / sb: -9900