[rrd-developers] interleaving RRD files, RRDfs perhaps?

[Daniel.Pocock at barclayscapital.com]

 > Normally the filesystem should be able to take care of quite 
> a lot of otimization duties as long as you present it with a 
> siutable workload. So maybe there is some low hanging fruit 
> to be harvested by looking at when and in what order 
> rrdcached does its updates and also maybe how they are 
> performed by the rrd update code, without gowing to a raw device ...
>

I'm certainly studying that as well - the interleaving would work at a
lower level than rrdcached, and would allow the block-level caching
systems to have a higher chance of success.
 
> I am pretty sure though that you can esily devise a much more 
> performant system if you start limiting the way data can be 
> deliverd (eg all rrds look the same)

That is certainly the case with some applications, e.g. Ganglia
> 
> Looking at the existing code, did you investigate creating 
> rrd files with a LOT of DSes ?

Yes, that seems like a possibility - but many applications, such as
Ganglia, create new RRDs on the fly.  Therefore, there would need to be
some way of creating an optimal number of empty DSes in advance, and
then mapping them to application-specific names when they are needed.
This logic could be implemented in the app or rrdtool.

Implementing within rrdtool in a way that preserves the existing rrdtool
API seems to be desirable - perhaps the different operating mode could
be enabled by some kind of environment variable and/or an rrdtool
configuration file in /etc?
_______________________________________________

This e-mail may contain information that is confidential, privileged or otherwise protected from disclosure. If you are not an intended recipient of this e-mail, do not duplicate or redistribute it by any means. Please delete it and any attachments and notify the sender that you have received it in error. Unless specifically indicated, this e-mail is not an offer to buy or sell or a solicitation to buy or sell any securities, investment products or other financial product or service, an official confirmation of any transaction, or an official statement of Barclays. Any views or opinions presented are solely those of the author and do not necessarily represent those of Barclays. This e-mail is subject to terms available at the following link: www.barcap.com/emaildisclaimer. By messaging with Barclays you consent to the foregoing.  Barclays Capital is the investment banking division of Barclays Bank PLC, a company registered in England (number 1026167) with its registered office at 1 Churchill Place, London, E14 5HP.  This email may relate to or be sent from other members of the Barclays Group.
_______________________________________________