[rrd-developers] implementing portable format

[Sfiligoi Igor]

kevin brintnall wrote:
> I have some more ideas on the implementation...  I tried to list the
> categories in increasing order of difficulty.  I'm sure I'm missing a few
> gotchas, but these strike me as the major categories that need work.
> 
> Looking for feedback on these...  Let me know if I'm off-base.
> 
> snip
> -----------------------------------------------------------------
> 
> PORTABLE VS. NATIVE FORMAT:
> 
> * Since we pass rrd_t around so many places, it's better if we have to
>   handle only a single type of struct in code.  When we rrd_open() an
>   older file, create a V0005 struct.  Keep the previous stat_head.version
>   so we can tell how to handle the file (i.e. whether we have to convert
>   values to/from native).
> 
> * If we keep the in-memory rrd_t.* in portable format, we have to convert
>   it in many places; some conversions are likely to get missed.  Instead,
>   we should convert it to native format in rrd_open.  Other code remains
>   largely unchanged.
> 
>   * the IN-FILE rrd_t header will be in portable format
>   * the IN-MEMORY rrd_t header will be in machine-native format
>   * therefore, we can't use the mmap()'ed version directly; we'll have
>     to copy+convert it
>   * in the reverse direction, we'll have to convert it back to portable
>     format and memcpy() on top of the mmap version.
Hi Kevin.

I can see performance issues with this.

Often only a small fraction of a RRD file is read.
With your proposal, the whole file needs to be loaded (and converted)
every time.

----

All other points seem very reasonable.

Cheers,
  Igor