[rrd-users] Re: Updating 1000's of rrd files

Tue Jul 6 03:26:20 MEST 2004

After thinking about it for five minutes, (which may cause me to be 
wrong... nonetheless...) 
With that many rrd files getting updated every 5 minutes, distribution and 
parallelism would really be the only way to go.

One solution is to have a cluster of "recorder" servers to handle all of 
the required writes with data sent from the set of distributed "collector" 
devices.  this cluster can be load balanced (i.e. ldir perhaps?) to 
distribute the load produced by all of the incoming requests.  For the 
storage you can abstract out this layer a little more and have a one or 
two terrabyte, uber-fast, NetApp handling the disks.

There really isn't an "easy", or "off-the-shelf" way to do this.  Lots of 
planning (and a mediocre amt. of money) is required to have a system as 
scalable as you indicate.

In my opinion, DB's would be a rockin' solution for this, however, with 
the automatic pruning of the data that RRD performs, and a few other 
reasons, it *could* be a little trickier, depending on the approach.  When 
you get to determine the back-end of this enterprise level of monitoring, 
you might just have six of one, half a dozen of the other.

my $0.02.  :)

Dan

"Jamie Lerner" <jlerner at jjlabs.com> 
Sent by: rrd-users-bounce at list.ee.ethz.ch
05/07/2004 08:26 PM

To
"Tobias Oetiker" <oetiker at ee.ethz.ch>, "Seth Hettich" 
<shettich at gmail.com>, "rrdtool-list" <rrd-users at list.ee.ethz.ch>
cc

Subject
[rrd-users] Re: Updating 1000's of rrd files

I too have the same problem, we are updating 40,000 - 120,000 rrd files
every five minutes and have found the following.

The systems become IO constrained quickly because each RRD file is opened,
edited then closed, which is very IO intensive for the OS.  You can do the
following to help speed it up:

RAID 1 plus 0 is much faster than RAID 5

Faster drives help (15,000 RPMs helps)

Memory helps (we use between 6 - 8 GB) but if you are looping every 5
minutes, the system can never catch up and eventually crashes anyway.

The real solution is to build some type of RRD server that can manage the
file IO and cacheing like a database does.  Is there any technology like
this for RRD files?  If not, we may consider moving from RRDs to a 
database
in order to get faster IO...something I would prefer not to do.

Jamie.

----- Original Message ----- 
From: "Tobias Oetiker" <oetiker at ee.ethz.ch>
To: "Seth Hettich" <shettich at gmail.com>
Cc: "rrdtool-list" <rrd-users at list.ee.ethz.ch>
Sent: Sunday, July 04, 2004 10:49 PM
Subject: [rrd-users] Re: Updating 1000's of rrd files

Yesterday Seth Hettich wrote:

> So, I have a problem:
>
> I need to collect some data from ~2,000 to 3,000 devices (perhaps more)
> and put it into rrd files.  about 100 files per device.  The code to
> collect all the data can run in ~2-3 min.  Also, code to update one
> rrd file ~5,000 times can run in ~20 sec.
>
> But, code to collect all the data and update the rrd files take over an
hour
>
> I assume this is due to the time to update so many files (in the test 
case
> it's just one file over and over, so it's very fast).
>
>
> I assume it would go much faster if I could put all of the data
> sources for each device in one file, but new data sources will get
> added all the time, and I see no way to add a DS to an rrd file.
>
> Anyone have any ideas for how to make this work?

In my tests when writing rrdtool I saw that the number of files you
can update efficiently is tighly coupled to two things:

a) it only works fast if you use local disk (nfs is normally very
   slow for this due to its 'write to disk first' behaviour

b) the number of files is tied to the amount of memory available
   for caching ... on our old sun server which I used for testing
   then, the rate of update dropped more than 10 fold once I went over
   700 files ... it went down to 20 updates a second ... These days
   everything will be faster, but I assume the basic problem remains
   ....

some systems also exhibit problems with having to deal with many
files in a single directory, so you might want to split this out
...

cheers
tobi

cheers
tobi

> -Seth
>
> --
> Unsubscribe mailto:rrd-users-request at list.ee.ethz.ch?subject=unsubscribe
> Help        mailto:rrd-users-request at list.ee.ethz.ch?subject=help
> Archive     http://www.ee.ethz.ch/~slist/rrd-users
> WebAdmin    http://www.ee.ethz.ch/~slist/lsg2.cgi
>

-- 
 ______    __   _
/_  __/_  / /  (_) Oetiker @ ISG.EE, ETZ J97, ETH, CH-8092 Zurich
 / // _ \/ _ \/ /  System Manager, Time Lord, Coder, Designer, Coach
/_/ \.__/_.__/_/   http://people.ee.ethz.ch/~oetiker   +41(0)44-632-5286

--
Unsubscribe mailto:rrd-users-request at list.ee.ethz.ch?subject=unsubscribe
Help        mailto:rrd-users-request at list.ee.ethz.ch?subject=help
Archive     http://www.ee.ethz.ch/~slist/rrd-users
WebAdmin    http://www.ee.ethz.ch/~slist/lsg2.cgi

--
Unsubscribe mailto:rrd-users-request at list.ee.ethz.ch?subject=unsubscribe
Help        mailto:rrd-users-request at list.ee.ethz.ch?subject=help
Archive     http://www.ee.ethz.ch/~slist/rrd-users
WebAdmin    http://www.ee.ethz.ch/~slist/lsg2.cgi

--
Unsubscribe mailto:rrd-users-request at list.ee.ethz.ch?subject=unsubscribe
Help        mailto:rrd-users-request at list.ee.ethz.ch?subject=help
Archive     http://www.ee.ethz.ch/~slist/rrd-users
WebAdmin    http://www.ee.ethz.ch/~slist/lsg2.cgi