[rrd-developers] interleaving RRD files, RRDfs perhaps?

[Tobias Oetiker]

Hi Daniel,

this has not been discussed yet. I guess the problem is that you
get into a region where there are not all that many people who have
installations so large that it matters.

Normally the filesystem should be able to take care of quite a lot
of otimization duties as long as you present it with a siutable
workload. So maybe there is some low hanging fruit to be harvested
by looking at when and in what order rrdcached does its updates and
also maybe how they are performed by the rrd update code, without
gowing to a raw device ...

I am pretty sure though that you can esily devise a much more
performant system if you start limiting the way data can be
deliverd (eg all rrds look the same)

Looking at the existing code, did you investigate creating rrd
files with a LOT of DSes ?

cheers
tobi



? then Today Daniel.Pocock at barclayscapital.com wrote:

>
>
>
> With a system such as Ganglia, there are many RRD files all created with
> exactly the same parameters.
>
> While rrdcached provides benefits by caching updates to each file, we
> still see lots of small 4kb random access writes to the SAN.
>
> Has anyone considered the possibility of aggregating multiple RRD files
> with identical RRAs into a single file, or even some kind of
> pseudo-filesystem?  This would mean that each disk block would contain
> data from many RRDs common to a specific update time.  There would be a
> dramatic improvement in write speed, as the OS and SAN would be caching
> the blocks relating to the current time period, rather than racing back
> and forth all over the disk.
>
> It would be particularly desirable to be able to tune the block size
> used for interleaving, so that it could maximise filesystem and SAN
> performance.
>
> Regards,
>
> Daniel
> _______________________________________________
>
> This e-mail may contain information that is confidential, privileged or otherwise protected from disclosure. If you are not an intended recipient of this e-mail, do not duplicate or redistribute it by any means. Please delete it and any attachments and notify the sender that you have received it in error. Unless specifically indicated, this e-mail is not an offer to buy or sell or a solicitation to buy or sell any securities, investment products or other financial product or service, an official confirmation of any transaction, or an official statement of Barclays. Any views or opinions presented are solely those of the author and do not necessarily represent those of Barclays. This e-mail is subject to terms available at the following link: www.barcap.com/emaildisclaimer. By messaging with Barclays you consent to the foregoing.  Barclays Capital is the investment banking division of Barclays Bank PLC, a company registered in England (number 1026167) with its registered off
 ic
>  e at 1 Churchill Place, London, E14 5HP.  This email may relate to or be sent from other members of the Barclays Group.
> _______________________________________________
>
> _______________________________________________
> rrd-developers mailing list
> rrd-developers at lists.oetiker.ch
> https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers
>
>

-- 
Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland
http://it.oetiker.ch tobi at oetiker.ch ++41 62 775 9902 / sb: -9900