[rrd-users] rrdcached issues with larger number of clients via network/pthread

[Tobias Oetiker]

Hi Ulf,

Today Ulf Zimmermann wrote:

> If I am running into memory, I am close to the top even shortly
> after restart:
>
> 29837 collectd  15   0 2990m  55m  772 S  8.0  0.3   1:44.38 rrdcached
>
> This is a 32-bit installation right now (I was going to go
> 64-bit, but had issue with .. rrdtool, although now with
> rrdcached I could get around that).
>
> 288 connected machines right now:
>
> log02 ulf /home/ulf $ netstat -an | grep 42217 | grep ESTA | wc -l
> 288

I would suggest you try the snapshot ... your problem seems simple
enough to reproduce, so you would see quickly if it helps ...

cheers
tobi


>
> > -----Original Message-----
> > From: Tobias Oetiker [mailto:tobi at oetiker.ch]
> > Sent: Sunday, October 31, 2010 12:10 AM
> > To: Ulf Zimmermann
> > Cc: 'rrd-users at lists.oetiker.ch'
> > Subject: Re: [rrd-users] rrdcached issues with larger number of clients
> > via network/pthread
> >
> > HI Ulf,
> >
> > Yesterday Ulf Zimmermann wrote:
> >
> > > I got close to 300 machines running collectd, configured to use
> > unixsocks to rrdcached on a central server. We are running more and
> > more into threads dieing (collectd then starts complaining and fills up
> > /var/messages) and when we try to restart collectd, sometimes it works,
> > sometimes we end up with:
> > >
> > > Oct 30 22:27:19 log02 rrdcached[16864]: listen_thread_main:
> > pthread_create failed.
> > > Oct 30 22:27:34 log02 rrdcached[16864]: listen_thread_main:
> > pthread_create failed.
> > > Oct 30 22:28:10 log02 rrdcached[16864]: listen_thread_main:
> > pthread_create failed.
> > >
> > > And at this point we usual have to restart the rrdcached daemon,
> > which then means having to restart collectd on close to 300 machines.
> > >
> > > How can this be debugged to find the issue (potential inside of
> > pthreads). The central server is running RedHat EL5 Update 4, the
> > rrdtool/rrdcached is 1.4.4 from rpmforge.
> > >
> > > Ulf, who is getting more grey hair by the minute with issues like
> > this :-(
> >
> > try the latest stable snapshot ... there are already a number of
> > fixes in the 1.4 branche ... for memory issues and such ... maybe
> > this affects your problem too.
> >
> > http://oss.oetiker.ch/rrdtool/pub/beta/rrdtool-1.4-svn-snap.tar.gz
> >
> > cheers
> > tobi
> >
> > >
> > > _______________________________________________
> > > rrd-users mailing list
> > > rrd-users at lists.oetiker.ch
> > > https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users
> > >
> > >
> >
> > --
> > Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland
> > http://it.oetiker.ch tobi at oetiker.ch ++41 62 775 9902 / sb: -9900
>
>

-- 
Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland
http://it.oetiker.ch tobi at oetiker.ch ++41 62 775 9902 / sb: -9900