[rrd-users] Re: Updating 1000's of rrd files

[Tobias Oetiker]

Yesterday Seth Hettich wrote:

> So, I have a problem:
>
> I need to collect some data from ~2,000 to 3,000 devices (perhaps more)
> and put it into rrd files.  about 100 files per device.  The code to
> collect all the data can run in ~2-3 min.  Also, code to update one
> rrd file ~5,000 times can run in ~20 sec.
>
> But, code to collect all the data and update the rrd files take over an hour
>
> I assume this is due to the time to update so many files (in the test case
> it's just one file over and over, so it's very fast).
>
>
> I assume it would go much faster if I could put all of the data
> sources for each device in one file, but new data sources will get
> added all the time, and I see no way to add a DS to an rrd file.
>
> Anyone have any ideas for how to make this work?

In my tests when writing rrdtool I saw that the number of files you
can update efficiently is tighly coupled to two things:

a) it only works fast if you use local disk (nfs is normally very
   slow for this due to its 'write to disk first' behaviour

b) the number of files is tied to the amount of memory available
   for caching ... on our old sun server which I used for testing
   then, the rate of update dropped more than 10 fold once I went over
   700 files ... it went down to 20 updates a second ... These days
   everything will be faster, but I assume the basic problem remains
   ....

some systems also exhibit problems with having to deal with many
files in a single directory, so you might want to split this out
...

cheers
tobi



cheers
tobi

> -Seth
>
> --
> Unsubscribe mailto:rrd-users-request at list.ee.ethz.ch?subject=unsubscribe
> Help        mailto:rrd-users-request at list.ee.ethz.ch?subject=help
> Archive     http://www.ee.ethz.ch/~slist/rrd-users
> WebAdmin    http://www.ee.ethz.ch/~slist/lsg2.cgi
>

-- 
 ______    __   _
/_  __/_  / /  (_) Oetiker @ ISG.EE, ETZ J97, ETH, CH-8092 Zurich
 / // _ \/ _ \/ /  System Manager, Time Lord, Coder, Designer, Coach
/_/ \.__/_.__/_/   http://people.ee.ethz.ch/~oetiker   +41(0)44-632-5286

--
Unsubscribe mailto:rrd-users-request at list.ee.ethz.ch?subject=unsubscribe
Help        mailto:rrd-users-request at list.ee.ethz.ch?subject=help
Archive     http://www.ee.ethz.ch/~slist/rrd-users
WebAdmin    http://www.ee.ethz.ch/~slist/lsg2.cgi