[rrd-developers] implementing portable format

[kevin brintnall]

On Thu, Oct 30, 2008 at 01:42:27PM -0500, Sfiligoi Igor wrote:
> > PORTABLE VS. NATIVE FORMAT:
> > 
> > * Since we pass rrd_t around so many places, it's better if we have to
> >   handle only a single type of struct in code.  When we rrd_open() an
> >   older file, create a V0005 struct.  Keep the previous stat_head.version
> >   so we can tell how to handle the file (i.e. whether we have to convert
> >   values to/from native).
> > 
> > * If we keep the in-memory rrd_t.* in portable format, we have to convert
> >   it in many places; some conversions are likely to get missed.  Instead,
> >   we should convert it to native format in rrd_open.  Other code remains
> >   largely unchanged.
> > 
> >   * the IN-FILE rrd_t header will be in portable format
> >   * the IN-MEMORY rrd_t header will be in machine-native format
> >   * therefore, we can't use the mmap()'ed version directly; we'll have
> >     to copy+convert it
> >   * in the reverse direction, we'll have to convert it back to portable
> >     format and memcpy() on top of the mmap version.

Igor,

> I can see performance issues with this.
> Often only a small fraction of a RRD file is read.

The RRD header is the only part that is read every time (this happens
already).  That is absolutely necessary to determine the RRD geometry
(ds_def, rra_def, rra_ptr, etc.).  The header is usually VERY small
compared to the rest of the file.

> With your proposal, the whole file needs to be loaded (and converted)
> every time.

I am only advocating that we read+convert the header in rrd_open; not the
entire file.  Then, we can deal with the header in native format.
Ultimately, the header is converted before being written back to the RRD
file.

As far as the values, RRDTool already read/writes only the relevant parts
of the file.  So, that's where we'd need to do the data conversion..  We
don't have to touch other parts of the file.

As Tobi already noted, the bit-conversions run at a rate that greatly
exceeds the overall processing rate of the RRD file.

-- 
 kevin brintnall =~ /kbrint at rufus.net/