[rrd-users] scaling RRDTool to the sky..and beyond

[nate]

hey folks,

 I've been using RRDTool for about 2 years now, and make
pretty extensive use of it these days. I'm not a programmer
so my abilities are limited but I am doing all the data
work using the RRDs perl module and rrdcgi for the graphing.

To make adding databases easily I store each data point in
it's own file. I am also interested in keeping every point
of data, I was never really able to figure out the RRAs so
I basically tell it to store everything for MIN/MAX/AVG/LAST,
which may be overkill, not sure.

anyways as I said I have a lot of RRD databases, around
15,000 at this point and they are updated at LEAST once per
minute. My servers struggle at around 80 updates per second.
Also I have around 300,000 lines of rrdcgi templates.

I am curious if anyone else out there runs largish scale
RRDs with lots of data points being collected with frequent
updates and what kind of hardware they have. I've found that
it's entirely I/O at this point. CPU stays 99% idle.

I have 8 servers, the most recent ones are the most powerful:

Dual 3Ghz Xeon 1MB cache
4GB memory
8x74GB 10k RPM Raptor disks in RAID10 (3ware raid)
7x74GB 10k RPM Raptor disks in RAID5 (3ware raid)
Debian Sarge(3.1) 2.4.31 kernel
(these servers are redundant, 2 systems monitoring the
same datapoints)

I get get as high as around 5,000 data updates a minute but
the box itself is almost unusable on the command line, it
can go 10-15 seconds while it waits on i/o. this doesn't
seem to adversely impact the generation of the graphs all that
much.

I'm using RRTool 1.0.42 I believe, I've been meaning to
upgrade but I have to go back and adjust all my cgis since
I use some options that cause newer versions(up to 1.0.48
at least) to generate errors, and with my frantic work
weeks i haven't had the time.

just lookin for stories/experiences with high load RRD
usage. Maybe there is something I can do to improve
performance without having to get a rack full of disks :)

I've already cut back on the amount of data points being
collected and the frequency of updates. At this rate it
is completely possible by this time next year I will
have upwards of 50,000 RRD databases.

How well does rrdtool work with NFS? some of my newer
systems will be backed by NetApps, I tried running
RRDTool over NFS(linux-to-linux) with full locking
enabled but it didn't work. it said it updated
successfully but all of the datapoints were 0. Didn't
spend much time researching why at the time. I don't
have any NetApps yet to test. All of my current systems
write to local disk.

thanks

nate
(I do plan to release my work to the world at some point
once I can get it cleaned up, maybe even write a good
HOWTO for RRDTool like I did with MRTG)

--
Unsubscribe mailto:rrd-users-request at list.ee.ethz.ch?subject=unsubscribe
Help        mailto:rrd-users-request at list.ee.ethz.ch?subject=help
Archive     http://lists.ee.ethz.ch/rrd-users
WebAdmin    http://lists.ee.ethz.ch/lsg2.cgi