[rrd-developers] patch/tuning for very large RRD systems (was "Re: SQL Backend request")

[Dave Plonka]

Hi Henrik,

On Tue, May 29, 2007 at 11:20:02PM +0200, Henrik Stoerner wrote:
> On Thu, May 24, 2007 at 01:43:56PM -0500, Dave Plonka wrote:
> > Note however, that such a configuration couldn't even do 100K
> > RRD files without the patch to fadvise RANDOM to suppress readahead.
> > I believe any post-2.6.5 Linux has the posix_fadvise behavior that
> > the patch leverages.
> > 
> > Also, Tobi has integrated the patch in the code he's testing.
> 
> I "only" have about 40K RRD's with 5-minute update intervals, but the
> machine had problems keeping up with this so I decided to try out your
> patches.
> 
> Since we're on RRDtool 1.0.49, though, I did a back-port of your 
> fadvise code to that version - very easy, but if anyone else
> wants it drop me a note and I'll put the diff up for download.
> 
> The result was that the memory utilisation for the buffer cache
> dropped dramatically, as I'd expected.

Rather than trying to guess what's going on in the buffer cache, we
could use fincore to determine how the RRD files are being cached:
   http://net.doit.wisc.edu/~plonka/fincore/

By running this periodically, one can see if things are going in the
right direction or if, for instance, RRD files are competing with
other things for buffer-cache.

> However, the general I/O
> activity did not go down - in fact, it increased by about 15-20%,
> as measured by vmstat.

With some more detail about what you saw, we could find what's
really going on.

reads or writes increased?  If performance was so poor before such
that it interfered with polling or graphing then it is possible for
overall I/O to go up afterwards.  Did the writes also increase or
just the reads?  By properly managing buffer-cache, it is possible
to get RRD reads to near zero.

Also, how long did it run before making this assessment (that I/O
increased)?

> This is a Linux 2.6.18 (Debian 4.0) system; the RRD files are on a
> ReiserFS filesystem.

I have no experience with reiser (just ext2 and ext3), but from
what I've read about it, I wouldn't expect the file system to be a
major performance concern here, since we really just want to use the
buffer-cache effectively.

> Re-coding the application that handles the RRD updates to cache them in
> memory and bundle updates together turned out to be a lot more effective 
> way of tackling the problem. We're running Hobbit, not MRTG - I have no
> idea if this would even be possible with MRTG.

It is, and we've done that as well.  From our research and
experimentation the two methods are about equally as effective, and
they are luckily complementary.  That is, the application caching
technique also benefits from the suppresses readahead that better
matches RRD random I/O.

My position on it is whether you have tens or low hundreds of thousands
of files, one can assemble a Linux server (perhaps with multiple
CPUs and Gigs of RAM) that can handle this level of performance.
If that is not possible (for financial or other reasons) and you find
that fadvise RANDOM alone results in insufficent performance on your
system, then techniques which change the I/O pattern dramatically by
application caching can be used in addition.

Dave

-- 
plonka at doit.wisc.edu  http://net.doit.wisc.edu/~plonka/  Madison, WI