[rrd-developers] Alternative to rrdcached on linux - rrdfs - FUSE based RRD caching

[Peter Stamfest]

Hello!

In the last couple of days, I implemented an alternative to the rrdcached: 
A FUSE based linux file system doing essentially the same as rrdcached.

The idea is as follows:

- There is a directory hierarchy (the permanent storage area), that holds 
RRD files.
- Another directory hierarchy is used to keep cached RRD files, I usually 
put this below /dev/shm (that is, into a memory based filesystem for 
extremely fast access), I call this the staging area.
- The rrdfs file system provides a layer above these directories, 
providing the same view as in permanent storage
- Updating RRDs is done by writing data to a special INPUT file in the 
format

  FILENAME <SPACE> <timestamp>:<ds-value>:<ds-value>:....

- Alternatively, data can be appended to the RRD file presented over 
rrdfs. In that case the FILENAME must not be written.
- A read access over rrdfs causes data to be updated to the staging area 
before handing RRD data to the reader
- Unlinking the RRD over rrdfs causes the staged data to be transferred 
back to permanent storage
- An optional cleanup thread causes the RRD updating and transfer from 
staging to permanent storage from time to time.
- plus a log-file is written in a write-ahead fashion. The log file gets 
rolled automatically by the cleanup thread.

I started the implementation only about a week ago, without knowing about 
rrdcached, so I was pretty embarassed when I found out about it. But I 
don't want this to be a complete waste of time. 

By the way: The thing is fully multithreaded.

So far I only tested it in a 1.2.x environment, 1.3 and 1.4 pending. The 
ONLY RRD API used is rrd_update_r, so I'm pretty sure there are no real 
hard problems to solve to get it to work with newer versions.

One nice feature is the immediate availability of RRD to any programming 
language that can append to files: no need to start a new process.

Downloads:

http://oss.stamfest.net/rrdfs/rrdfs-1.0.tar.gz

Mercurial repo available on request.

I would be willing to actually have this distributed alongside of rrdtool 
proper. But as I said: It is an alternative to rrdcached and it only works 
on linux. I'd be happy to actually hear thoughts about the usefulness of 
this beast. Benchmarking results would also be great.

peter


_________________________________________________________________________
Dipl.-Ing. Peter Stamfest     UNIX, Networking & Computing Consultant
Tel: +43/699/10711205         Software Development - Internetservices
E-Mail: peter at stamfest.at     WWW: http://stamfest.at/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.oetiker.ch/pipermail/rrd-developers/attachments/20100223/769085de/attachment.htm