[rrd-users] Re: Who has the most RRD files (or data sources)?

[nate]

Mark Plaksin said:
> Who has the most RRD files (or total data sources in RRDs)?  How frequently
> are they updated?  What do you use to update them?


At my last company(just left it) I made a custom RRD system that
had about 20-30k of RRD files spread accross 8 servers, each
datafile was updated at least once per minute, there was about
5k that were updated more often than that(ranging from 5 seconds
to 45 seconds). The servers were taxed quite hard, the systems
were not very responsive on the command line. system specs for
the latest round of systems was:

Dual 3Ghz Xeon 1MB cache
4GB memory
8x74GB 10k RPM Raptor disks in RAID10 (3ware raid)
7x74GB 10k RPM Raptor disks in RAID5 (3ware raid)
Debian Sarge(3.1) 2.4.31 kernel
(these servers are redundant, 2 systems monitoring the
same datapoints)

(these systems struggled with disk I/O with about 5k files updating
every minute, load was typically in the 5.0-7.0 range, didn't appear
to impact end user experience using rrdcgi but using the shell was
quite painful)

>
> We are using a home-grown system to update over 20,000 RRD files.  Each RRD
> file has a single data source.  RRD files are using a total of 45G of disk
> space.

my datafiles consumed around 1.3TB of space I think. My standard
was to keep 2 years worth of data so the typical datafile size was
around 65MB. I am sure it wasn't the best way to store the data but
I couldn't find any real clear easy to understand docs on how to
store stuff in RRAs so I just stored every data point and never
went back and re-evaluated things. I created:

RRA:(MIN|AVG|MAX|LAST):0.5:1:1051200


> Clearly we're doing something wrong :)  I'm looking for pointers about
> making our system more efficient.  While it is possible that our machine
> has a poor IO system, let's pretend it doesn't and just talk about the most
> efficient way to get data into a large number of RRD files.  We're
> investigating the IO system separately.
>
> Besides the size of the RRD files, does the number of rows per RRA impact
> performance?

I asked a similar question a few months back (9/17/2005 - "scaling RRDTool
to the sky..and beyond") and did not get any responses(that I noticed). What
sort of hardware are you running?

nate

--
Unsubscribe mailto:rrd-users-request at list.ee.ethz.ch?subject=unsubscribe
Help        mailto:rrd-users-request at list.ee.ethz.ch?subject=help
Archive     http://lists.ee.ethz.ch/rrd-users
WebAdmin    http://lists.ee.ethz.ch/lsg2.cgi