[rrd-developers] Alternative to rrdcached on linux - rrdfs - FUSE based RRD caching

[Peter Stamfest]

Tobias Oetiker <tobi at oetiker.ch> schrieb am 23.02.2010 23:28:53:

> Hi Peter,
> 
> wow ... sounds impressive. Have you done performance evaluations ?
> 

Only insofar, as it is fast. Accessing RRDs in /dev/shm is just "wow". I 
wanted to do my first real deployment of the software yesterday, and then 
I stumbled across rrdcached. And my motivation just vanished :-(

> If you could apply your genius to rrdcached and maybe make it even
> better based on the insights gained from the rrdfs implementation?

Well, I had some plans for rrdfs: 

- Provide a memory only version of rrd_update_r. Then I could get rid of 
the staging area and just cache RRDs entirely in memory - while still 
providing file-level access to them. 
- The write ahead logging rrdfs is able to do would allow for proper 
restart of a crashed/killed/whatever rrdfs in such a scenario (but it 
would also help for the staging-area case).
- Make it NFS capable: Access RRDs handled by rrdfs from a remote host 
using normal file access. This requires a recent kernel and has some 
drawbacks, though. But this would allow for HUGE RRD repositories - 
distributed over any number of NFS servers.
- It would even  be possible to integrate more of rrdtool commands into 
rrdfs. Just imagine a mechanism to just access a png file by name and have 
it automagically computed from memory. It would even be possible to 
provide "pseudo" directories for every RRD, eg. containing RRD data as CSV 
"files". The sky is the limit (or the avoidance of feature creep)...


I don't know the inner workings of rrdcached, but rrdfs could actually 
become part of it, I think (#ifdef'd on the availability of FUSE). The 
question is: Is this something you would accept?


So a plan could be:
 - factor out file access from rrd_update_r and provide a rrd_update_mem_r 
as a base for rrd_update_r and become memory only (with the transparent 
permanent storage handling).
 - integrate the IPC handling from rrdcached into rrdfs or vice-versa. 
This MIGHT be as simple as just modifying the main() from rrdfs to fire up 
a single pthread and have that take care of rrdfs and all the rest would 
be handled by the existing rrdcached (plus the backend storage/update 
caching would have to be integrated - in that area I'm using a hashtable 
keyed on the file name essentially containing a dynamic array of updates 
per file).

peter



> 
> Today Peter Stamfest wrote:
> 

[.. removed ..]


_________________________________________________________________________
Dipl.-Ing. Peter Stamfest     UNIX, Networking & Computing Consultant
Tel: +43/699/10711205         Software Development - Internetservices
E-Mail: peter at stamfest.at     WWW: http://stamfest.at/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.oetiker.ch/pipermail/rrd-developers/attachments/20100224/7eef968e/attachment.htm