[rrd-users] Re: Updating 1000's of rrd files

[Jamie Lerner]

I too have the same problem, we are updating 40,000 - 120,000 rrd files
every five minutes and have found the following.

The systems become IO constrained quickly because each RRD file is opened,
edited then closed, which is very IO intensive for the OS.  You can do the
following to help speed it up:

RAID 1 plus 0 is much faster than RAID 5

Faster drives help (15,000 RPMs helps)

Memory helps (we use between 6 - 8 GB) but if you are looping every 5
minutes, the system can never catch up and eventually crashes anyway.

The real solution is to build some type of RRD server that can manage the
file IO and cacheing like a database does.  Is there any technology like
this for RRD files?  If not, we may consider moving from RRDs to a database
in order to get faster IO...something I would prefer not to do.

Jamie.


----- Original Message ----- 
From: "Tobias Oetiker" <oetiker at ee.ethz.ch>
To: "Seth Hettich" <shettich at gmail.com>
Cc: "rrdtool-list" <rrd-users at list.ee.ethz.ch>
Sent: Sunday, July 04, 2004 10:49 PM
Subject: [rrd-users] Re: Updating 1000's of rrd files


Yesterday Seth Hettich wrote:

> So, I have a problem:
>
> I need to collect some data from ~2,000 to 3,000 devices (perhaps more)
> and put it into rrd files.  about 100 files per device.  The code to
> collect all the data can run in ~2-3 min.  Also, code to update one
> rrd file ~5,000 times can run in ~20 sec.
>
> But, code to collect all the data and update the rrd files take over an
hour
>
> I assume this is due to the time to update so many files (in the test case
> it's just one file over and over, so it's very fast).
>
>
> I assume it would go much faster if I could put all of the data
> sources for each device in one file, but new data sources will get
> added all the time, and I see no way to add a DS to an rrd file.
>
> Anyone have any ideas for how to make this work?

In my tests when writing rrdtool I saw that the number of files you
can update efficiently is tighly coupled to two things:

a) it only works fast if you use local disk (nfs is normally very
   slow for this due to its 'write to disk first' behaviour

b) the number of files is tied to the amount of memory available
   for caching ... on our old sun server which I used for testing
   then, the rate of update dropped more than 10 fold once I went over
   700 files ... it went down to 20 updates a second ... These days
   everything will be faster, but I assume the basic problem remains
   ....

some systems also exhibit problems with having to deal with many
files in a single directory, so you might want to split this out
...

cheers
tobi



cheers
tobi

> -Seth
>
> --
> Unsubscribe mailto:rrd-users-request at list.ee.ethz.ch?subject=unsubscribe
> Help        mailto:rrd-users-request at list.ee.ethz.ch?subject=help
> Archive     http://www.ee.ethz.ch/~slist/rrd-users
> WebAdmin    http://www.ee.ethz.ch/~slist/lsg2.cgi
>

-- 
 ______    __   _
/_  __/_  / /  (_) Oetiker @ ISG.EE, ETZ J97, ETH, CH-8092 Zurich
 / // _ \/ _ \/ /  System Manager, Time Lord, Coder, Designer, Coach
/_/ \.__/_.__/_/   http://people.ee.ethz.ch/~oetiker   +41(0)44-632-5286

--
Unsubscribe mailto:rrd-users-request at list.ee.ethz.ch?subject=unsubscribe
Help        mailto:rrd-users-request at list.ee.ethz.ch?subject=help
Archive     http://www.ee.ethz.ch/~slist/rrd-users
WebAdmin    http://www.ee.ethz.ch/~slist/lsg2.cgi

--
Unsubscribe mailto:rrd-users-request at list.ee.ethz.ch?subject=unsubscribe
Help        mailto:rrd-users-request at list.ee.ethz.ch?subject=help
Archive     http://www.ee.ethz.ch/~slist/rrd-users
WebAdmin    http://www.ee.ethz.ch/~slist/lsg2.cgi