[rrd-users] rrdcached issues with larger number of clients via network/pthread

[Ulf Zimmermann]

Finally built a updated version into /opt/rrdtool-1.4.4.002147, but it seems it still growing when it comes to memory use.

Initial start up it was at 2,776MB virtual size, has grown to 3,000MB now in 3 days.


> -----Original Message-----
> From: Tobias Oetiker [mailto:tobi at oetiker.ch]
> Sent: Sunday, October 31, 2010 2:44 AM
> To: Ulf Zimmermann
> Cc: 'rrd-users at lists.oetiker.ch'
> Subject: RE: [rrd-users] rrdcached issues with larger number of clients
> via network/pthread
> 
> Hi Ulf,
> 
> Today Ulf Zimmermann wrote:
> 
> > If I am running into memory, I am close to the top even shortly
> > after restart:
> >
> > 29837 collectd  15   0 2990m  55m  772 S  8.0  0.3   1:44.38
> rrdcached
> >
> > This is a 32-bit installation right now (I was going to go
> > 64-bit, but had issue with .. rrdtool, although now with
> > rrdcached I could get around that).
> >
> > 288 connected machines right now:
> >
> > log02 ulf /home/ulf $ netstat -an | grep 42217 | grep ESTA | wc -l
> > 288
> 
> I would suggest you try the snapshot ... your problem seems simple
> enough to reproduce, so you would see quickly if it helps ...
> 
> cheers
> tobi
> 
> 
> >
> > > -----Original Message-----
> > > From: Tobias Oetiker [mailto:tobi at oetiker.ch]
> > > Sent: Sunday, October 31, 2010 12:10 AM
> > > To: Ulf Zimmermann
> > > Cc: 'rrd-users at lists.oetiker.ch'
> > > Subject: Re: [rrd-users] rrdcached issues with larger number of
> clients
> > > via network/pthread
> > >
> > > HI Ulf,
> > >
> > > Yesterday Ulf Zimmermann wrote:
> > >
> > > > I got close to 300 machines running collectd, configured to use
> > > unixsocks to rrdcached on a central server. We are running more and
> > > more into threads dieing (collectd then starts complaining and
> fills up
> > > /var/messages) and when we try to restart collectd, sometimes it
> works,
> > > sometimes we end up with:
> > > >
> > > > Oct 30 22:27:19 log02 rrdcached[16864]: listen_thread_main:
> > > pthread_create failed.
> > > > Oct 30 22:27:34 log02 rrdcached[16864]: listen_thread_main:
> > > pthread_create failed.
> > > > Oct 30 22:28:10 log02 rrdcached[16864]: listen_thread_main:
> > > pthread_create failed.
> > > >
> > > > And at this point we usual have to restart the rrdcached daemon,
> > > which then means having to restart collectd on close to 300
> machines.
> > > >
> > > > How can this be debugged to find the issue (potential inside of
> > > pthreads). The central server is running RedHat EL5 Update 4, the
> > > rrdtool/rrdcached is 1.4.4 from rpmforge.
> > > >
> > > > Ulf, who is getting more grey hair by the minute with issues like
> > > this :-(
> > >
> > > try the latest stable snapshot ... there are already a number of
> > > fixes in the 1.4 branche ... for memory issues and such ... maybe
> > > this affects your problem too.
> > >
> > > http://oss.oetiker.ch/rrdtool/pub/beta/rrdtool-1.4-svn-snap.tar.gz
> > >
> > > cheers
> > > tobi
> > >
> > > >
> > > > _______________________________________________
> > > > rrd-users mailing list
> > > > rrd-users at lists.oetiker.ch
> > > > https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users
> > > >
> > > >
> > >
> > > --
> > > Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten,
> Switzerland
> > > http://it.oetiker.ch tobi at oetiker.ch ++41 62 775 9902 / sb: -9900
> >
> >
> 
> --
> Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland
> http://it.oetiker.ch tobi at oetiker.ch ++41 62 775 9902 / sb: -9900