[rrd-developers] implementing portable format

[kevin brintnall]

On Thu, Oct 30, 2008 at 11:53:07PM +0100, Tobias Oetiker wrote:
> I would use the amd64/linux format since this will cover the 64bit
> alignement issue you mention below as well.
> I would just pick the one that goes with amd64

OK.

> I would like to see full support (read/write. not necessarily
> create). the motivation is as follows.

That's probably the right approach.  As long as we can avoid spreading the
logic over many places in the code, it should be OK.

> * thinking about the work dan is doing I think it would be an
>   overall good thing to find a sensible design for rrd_open which
>   de-coules internal and on-disk representation of rrd data to a
>   large extent.

rrd_open() can return a valid rrd_t header, and rrd_close() can write the
header back out to the file.  We can encapsulate all the header logic
along with those two.

Maybe we can create a similar method for accessing data values that
encapsulates the various storage backends...  Right now, we just base it
on offset relative to rra_start.

> * adoption will be much quicker if we do not force people to re-do
>   all their rrd files and maybe even destroy them in the process.

agreed.

> >   ==> We can determine byte encoding at build time.  We won't need a
> >       catalog of architectures and associated conversion macros.
> 
> this would be cool ... although, I think we have to have a solid
> set of test to make sure an automatically generated convertor
> actually does what it is suposed todo. after all it would be a big
> desaster if rrdtool on a certain platform just generated
> non-portable-portable files ...

Agreed...  I envision building the test cases into the macro-generator..
If it doesn't pass the tests, we can abort the build verbosely with enough
information to send back to the developers, etc.

> > * I'm thinking something analogous to the nthos()/htons() functions that
> >   can be used to convert each data type from native-format to
> >   portable-format.  i.e. htorrd_d(double), htorrd_i(int64_t) or something
> >   similar.
> 
> this would then be used in rrd_open ...

and also with the data values...

> > * create utility functions to simplify...  optionally they can determine
> >   whether conversion is necessary based on stat_head.version.
> >      - read and convert-to-native (double/int64_t)
> >      - convert-to-portable and write (double/int64_t)
> 
> this could be macros too

true.. it depends on how complicated they get.

> when thinking about the new format, it might also be worth spending
> some thought on shortcomings of the present format. I think it
> might be possible to add the following features quite easliy
> 
> * unlimited string length for labels
> With unlimited I mean that it is parametrized and gets set at rrd
> creation time.

Have there been many cases where the label length is not sufficient?

> * ability to store 'user data' in the rrd header and along with each
> datasource

For example?

> Also the sizes on scratch space in PDP and CPD areas could be
> paramtetrized which would help with future enhancements of the data
> format.

Why would they need to be variable size?  I'm a little fuzzy on the
inner-workings of [pc]dp_prep.

-- 
 kevin brintnall =~ /kbrint at rufus.net/