[rrd-developers] patch/tuning for very large RRD systems (was "Re: SQL Backend request")

[Dave Plonka]

Hi Tobi,

This is one of those serendipitous occasions -
I think we can save you perhaps lots of time:

Archit Gupta, Dale Carder, and I have just comleted a research project
on RRD performance and scalability.  I beleive we may have the largest
single system MRTG w/RRD - over 300,000 targets and RRD files updated
every five minutes.

On Wed, May 16, 2007 at 10:00:12AM +0200, Tobias Oetiker wrote:
> 
> I am still gathering data ... but it seems all to come down to
> 
> * file system cache pollution through other processes
> * and the time you give the system to deal with dirty bufffers
> * the block queuse size may also play a role.
> * I think we could gain quite a lot by using fadvise in rrdtool to
>   only keep the header portion of the file in cache.
>   will try this later today ...

Over the past month, we've done a posix_fadvise RANDOM patch and
completely evaluated the perforance impacts.  It's really good
(obviously).  What it does is get both the readahead and page faults
under control, in Linux at least.

For others observing performance issues, the pertinent things are:

 * What operating system and version are you running?
   (to determine initial file readahead and availablity of fadvise syscall)

 * What is your hardware, including physical memory?
   to determine CPU available and max buffer-cache size

* What is your update interval (rrd step) and RRD file definitions (RRAs)?
  (e.g. typical MRTG, or output of rrdtool info.)

  This determines the page fault characteristics of RRD files
  when buffer-cache is scarce.

* What version of rrdtool?
  (for applying patch)

With that we can determine if it's possible to update a givent number
of RRD files, or how to properly (re)size it.

Dave

-- 
plonka at doit.wisc.edu  http://net.doit.wisc.edu/~plonka/  Madison, WI
-------------- next part --------------
--- rrdtool-1.0.49/src/rrd_open.c_orig	2004-08-08 05:58:14.000000000 -0500
+++ rrdtool-1.0.49/src/rrd_open.c	2007-04-13 16:23:44.000000000 -0500
@@ -10,8 +10,10 @@
  *
  *****************************************************************************/
 
+#define _POSIX_C_SOURCE 200112L /* for posix_fadvise */
 #include "rrd_tool.h"
 #define MEMBLK 8192
+#include <fcntl.h> /* for posix_fadvise */
 
 /* open a database file, return its header and a open filehandle */
 /* positioned to the first cdp in the first rra */
@@ -41,6 +43,12 @@
 	rrd_set_error("opening '%s': %s",file_name, strerror(errno));
 	return (-1);
     }
+
+    /* on Linux 2.6.9, POSIX_FADV_RANDOM supresses read-ahead */
+    if (0 != posix_fadvise(fileno(*in_file), 0, 0, POSIX_FADV_RANDOM)) {
+       fclose(*in_file);
+       return(-1);
+    }
     
 #define MYFREAD(MYVAR,MYVART,MYCNT) \
     if ((MYVAR = malloc(sizeof(MYVART) * MYCNT)) == NULL) {\