[rrd-developers] SQL Backend request

[Tobias Oetiker]

Hi Martin,

I am still gathering data ... but it seems all to come down to

* file system cache pollution through other processes
* and the time you give the system to deal with dirty bufffers
* the block queuse size may also play a role.
* I think we could gain quite a lot by using fadvise in rrdtool to
  only keep the header portion of the file in cache.
  will try this later today ...

cheers
tobi

Today Martin Sperl wrote:

> Hi Toby!
>
> Do you know which parameters you are talking about exactly?
>
> We have experience that for some reason we are able to update 40k rrd files
> every 5 minutes on an Intel RH7.2 system, but when we tried to migrate to
> RHES4 the system would no longer perform and we reach a limit of 20k RRD files
> updated every 5 minutes and then the updates to all those files fall apart.
>
> It seems as if the 2.6 Kernel is doing some more agressive prefetching of
> blocks when opening each RRD file...
>
> I remember having looked for tweaking, but I have never found a conclusive
> answer... That is why I initially started the libdbi patch...
>
> Ciao,
>          Martin
>
> Tobias Oetiker wrote:
> > Hi Robert,
> >
> > you may want to look at your sytem configuration ... especially
> > blocksized and cache handling ... how often to you want to update
> > these 100k rrd files per minute  ?
> >
> > cheers
> > tobi
> >
> >
> >
> > > I while ago I noticed a topic that let rrdtool use a database backend.
> > > What
> > > is the status with this project?  Also, has this turned out to be a viable
> > > solution when updating large amounts of rrd files?  We're currently
> > > exploring other storage media to help our writes become faster.  We are
> > > currently writing just under 100K rrd files and have distributed writing.
> > > We were thinking of modifying rrdtool to use a MySql database and would
> > > like
> > > to modify the database beackend I saw on the list (if needed) accordingly
> > > so
> > > that we don't have to re-invent the wheel.
> > >
> > > I tried to look through the archives, but was only seeing patches to the
> > > library file.  Is there a place to get the whole thing at?  I couldn't see
> > > it in the /lib or /contrib download directories of the main site.
> > >
> > >
> > >
> >
> >
>
>

-- 
Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten
http://it.oetiker.ch tobi at oetiker.ch ++41 62 213 9902