[rrd-developers] patch/tuning for very large RRD systems (was "Re: SQL Backend request")

[Steve Hill]

On May 24, 2007, at 12:51 PM, Mark Plaksin wrote:

> We have about 45k RRDs and our testing so far says the fadvise changes
> are very nice--thanks!  We're also testing local disk (via cciss driver)
> vs SAN storage.  Our current RRD server is pretty crushed io-wise.  So
> far the SAN storage looks like a big win too.

We have a system here with about 65k RRDs, again updated every 5 minutes.
I will definitely have to take a look at the fadvise changes, since our
system is currently WAAY overloaded.

One thing I have noticed is the periodicity of the load. All our RRDs
have the same spec. The system happily churns away until it passes one
of the larger consolidation points and then *bam* the load goes through
the roof because of the extra IO.

For our system, I am interested in a way of smoothing out those load
spikes, since the system becomes periodically unusable. Yes, I do
appreciate that our updater could be smarter :)

One of the things I thought about was adding some offset to each RRD
such that the consolidation didn't happen on all RRDs at once, perhaps
based on the creation time of the RRD...

Comments?

Steve