[rrd-developers] Alternative to rrdcached on linux - rrdfs - FUSE based RRD caching

[Tobias Oetiker]

Hi Peter,

Today Peter Stamfest wrote:

> Tobias Oetiker <tobi at oetiker.ch> schrieb am 23.02.2010 23:28:53:
>
> > Hi Peter,
> >
> > wow ... sounds impressive. Have you done performance evaluations ?
> >
>
> Only insofar, as it is fast. Accessing RRDs in /dev/shm is just "wow". I
> wanted to do my first real deployment of the software yesterday, and then
> I stumbled across rrdcached. And my motivation just vanished :-(
>
> > If you could apply your genius to rrdcached and maybe make it even
> > better based on the insights gained from the rrdfs implementation?
>
> Well, I had some plans for rrdfs:
>
> - Provide a memory only version of rrd_update_r. Then I could get rid of
> the staging area and just cache RRDs entirely in memory - while still
> providing file-level access to them.
> - The write ahead logging rrdfs is able to do would allow for proper
> restart of a crashed/killed/whatever rrdfs in such a scenario (but it
> would also help for the staging-area case).
> - Make it NFS capable: Access RRDs handled by rrdfs from a remote host
> using normal file access. This requires a recent kernel and has some
> drawbacks, though. But this would allow for HUGE RRD repositories -
> distributed over any number of NFS servers.
> - It would even  be possible to integrate more of rrdtool commands into
> rrdfs. Just imagine a mechanism to just access a png file by name and have
> it automagically computed from memory. It would even be possible to
> provide "pseudo" directories for every RRD, eg. containing RRD data as CSV
> "files". The sky is the limit (or the avoidance of feature creep)...
>
>
> I don't know the inner workings of rrdcached, but rrdfs could actually
> become part of it, I think (#ifdef'd on the availability of FUSE). The
> question is: Is this something you would accept?
>
>
> So a plan could be:
>  - factor out file access from rrd_update_r and provide a rrd_update_mem_r
> as a base for rrd_update_r and become memory only (with the transparent
> permanent storage handling).
>  - integrate the IPC handling from rrdcached into rrdfs or vice-versa.
> This MIGHT be as simple as just modifying the main() from rrdfs to fire up
> a single pthread and have that take care of rrdfs and all the rest would
> be handled by the existing rrdcached (plus the backend storage/update
> caching would have to be integrated - in that area I'm using a hashtable
> keyed on the file name essentially containing a dynamic array of updates
> per file).

well I guess doing fuse based stuff is funky but since rrdtool is
used on a wide variety of OSes I am not as enthusiastic about it as
I would be about our crossplatfom rrdcached :-) ... eventhough the
interface is different, the functionality is pretty similar ...
what is missing are the following things

* authentication/authorization and encryption for network links
* in-memory updates, so that functions like updatev would work too.

when it comes to in-memory updates there is probably a tie-in with
the planned move from a machine specific data format to a
cross-platform dataformat which will require some additional
abstraction layer anyway.

cheers
tobi


>
> peter
>
>
>
> >
> > Today Peter Stamfest wrote:
> >
>
> [.. removed ..]
>
>
> _________________________________________________________________________
> Dipl.-Ing. Peter Stamfest     UNIX, Networking & Computing Consultant
> Tel: +43/699/10711205         Software Development - Internetservices
> E-Mail: peter at stamfest.at     WWW: http://stamfest.at/
>

-- 
Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland
http://it.oetiker.ch tobi at oetiker.ch ++41 62 775 9902 / sb: -9900