[rrd-users] rrdcached performance with >200k nodes

[Mirek Lauš]

Hello list,

we've probably reached rrdcached limits in our monitoring system

We had a very nicely running rrdcached while collecting from about 400 hosts,
about 100k nodes (RRD files).

We've bumped the number of host to about 2000 hosts for interface
traffic, errors, unicast and multicast packets with collector of our
own. It does batch the RRD updates using rrdcached's BATCH via unix
socket. This collector is able to walk
all the hosts in less than 5 minutes. The number of nodes is about 200k.

The rrdcached is configured to -w 3600 -z 3600 -f 7200 -t 8. Everything runs
smoothly until first timeout. Then the Queue value rises up to the
number of nodes
and keeps that high. Write rate is very low, disk IO is almost zero.
CPU load done by rrdcached gets very high (100-200%).

The system is FreeBSD 7.2-p4, amd64 with 16GB RAM, RAID10 disk array.
rrdtool 1.4.2.

Could it be we've reached rrdcached's limits? What can be done about it?

Kind regards,
Miroslav