[rrd-developers] Re: How to get the most performance when using lots of RRD files
tarus at opennms.org
Wed Aug 16 17:09:54 MEST 2006
Ole Bjørn Hessen wrote:
> Another solution is to make the RRD-files smaller :-)
We found the exact opposite to be true. We have a couple of clients who
are updating 500K+ values every five minutes. We used to store each
value in its own RRD file, but found that if we group them together
things run much faster.
NetApps rule. Move your storage off to one of their devices, hook up a
dedicated gigabit Ethernet interface, and watch the speed increase.
If you don't have/can't get a NetApp or similar device, more spindles
rule. Avoid RAID-5 like the plague - we use RAID 1+0 with lots of write
cache, usually with four disks.
The final thing that we did was abstract data collection from data
storage. Things would be going fine, and then I'd do something like a
"find" to see how many files we were updating, and bam, chaos, the
raptors were out, everything went sour.
By adding a queue to store values waiting to be written, we were able to
absorb such file system shocks. It takes just about as much time to
write two sets of values as one (most of the overhead is in the headers)
and so we can catch up. In fact, the further behind we get, the faster
we catch up. (grin)
The OpenNMS Group, Inc.
Main : +1 919 533 0160 Fax: +1 503-961-7746
Direct: +1 919 647 4749 Skype: tarusb
Key Fingerprint: 8945 8521 9771 FEC9 5481 512B FECA 11D2 FD82 B45C
Unsubscribe mailto:rrd-developers-request at list.ee.ethz.ch?subject=unsubscribe
Help mailto:rrd-developers-request at list.ee.ethz.ch?subject=help
More information about the rrd-developers