[rrd-developers] Re: How to get the most performance when using lots of RRD files

[Tarus Balog]

Ole Bjørn Hessen wrote:

> Another solution is to make the RRD-files smaller :-)

We found the exact opposite to be true. We have a couple of clients who
are updating 500K+ values every five minutes. We used to store each
value in its own RRD file, but found that if we group them together
things run much faster.

NetApps rule. Move your storage off to one of their devices, hook up a
dedicated gigabit Ethernet interface, and watch the speed increase.

If you don't have/can't get a NetApp or similar device, more spindles
rule. Avoid RAID-5 like the plague - we use RAID 1+0 with lots of write
cache, usually with four disks.

The final thing that we did was abstract data collection from data
storage. Things would be going fine, and then I'd do something like a
"find" to see how many files we were updating, and bam, chaos, the
raptors were out, everything went sour.

By adding a queue to store values waiting to be written, we were able to
absorb such file system shocks. It takes just about as much time to
write two sets of values as one (most of the overhead is in the headers)
and so we can catch up. In fact, the further behind we get, the faster
we catch up. (grin)

HTH,
-T
-- 

Tarus Balog
The OpenNMS Group, Inc.
Main  : +1 919 533 0160   Fax:   +1 503-961-7746
Direct: +1 919 647 4749   Skype: tarusb
Key Fingerprint: 8945 8521 9771 FEC9 5481  512B FECA 11D2 FD82 B45C

--
Unsubscribe mailto:rrd-developers-request at list.ee.ethz.ch?subject=unsubscribe
Help        mailto:rrd-developers-request at list.ee.ethz.ch?subject=help
Archive     http://lists.ee.ethz.ch/rrd-developers
WebAdmin    http://lists.ee.ethz.ch/lsg2.cgi