[rrd-developers] interleaving RRD files, RRDfs perhaps?

[Tobias Oetiker]

Hi Daniel,

Today Daniel.Pocock at barclayscapital.com wrote:

>
> > I think this leads to the rrd as a dbms discussion we touched
> > a few days ago on the list. This would open up up many
> > possibilities but also have to potential of makeing things a
> > lot more complex. In my eyes it is primarily a matter of
> > finding a really clever design for something like this.
> >
>
> It would be nice if rrdtool had some special way of interpreting the
> filenames and mapping them to some kind of heirarchical dbms structure.
> Then the application wouldn't need to know anything about what was going
> on under the hood.
>
> E.g. when the user does `rrdcreate /hostA/eth0_in.rrd ...', rrdtool
> would:
>
> - find a real file on disk containing DSes of the same format (same
> step, same xff, etc), or create such a file with a predefined number of
> empty DSes if it doesn't already exist
>
> - make a record in some database associating /hostA/eth0_in with the
> real filename and the internally assigned DS index
>
> - subsequent calls to rrdupdate on /hostA/eth0_in.rrd would be mapped,
> behind the scenes, to the real DS
>
> Maybe the filenames could be given a URL-style prefix to indicate that
> they are virtual?  E.g. rrd:/hostA/eth0_in.rrd
>
> If they are just multiple DSes in a normal RRD file, there is a problem
> of how to ensure they are all updated simultaneously.  However, if this
> was an enhanced file format, then the DSes could have independent update
> times - even if they were not always updated together, or even if they
> are being updated by separate rrdupdate processes called from shell
> scripts, the OS level write cache would still have a good chance of
> doing it's job.

the problem is that at some point rrdtool MUST write the RRA and as
soon as it does that, there is no going back. It all becomes pretty
hairy quickly if you aim for the general case, which you have to if
you are looking at integrating this functionality into the core ...

Personally I would try to implement such a scheme at the
application layer and only once I have used it for some time and
have gained deep insight into the nature of the issues that
surround this type of solution, I would start thinking about
generalizing it to the point where it becomes generally usefull.

cheers
tobi

-- 
Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland
http://it.oetiker.ch tobi at oetiker.ch ++41 62 775 9902 / sb: -9900