[rrd-users] Architecture independence for rrd files

[Steve Shipway]

>> 1. endian - is it MSB first or LSB first (or middle byte first as in windows)
>> 2. 32/64 bit - how big is an integer?  This adjusts the size of the struct
>> 3. alignment - does your architecture demand integers be byte-aligned and therefore generate padding bytes?
>> 4. compiler peculiarities - some just do it differently at compile time.
>
>To be honest, I can't see why any of what you have mentioned above are
>an issue. That is if it was catered for right at the beginning.
>Putting a layer between storage and working data is rather common
>practice to prevent exactly that type of problem.
>
>Things like how big is an integer, is rather irrelevant if you use
>integer types that are well defined (eg. int16_t, in32_t, in64_t vs
>plain int) Alignment is only an issue at runtime, not storage.

I'm only guessing at what was in Tobi's mind when he designed it, but I think the idea was to sacrifice portability in exchange for speed (and maybe the idea of architecture independence didn't occur to him back in the 90s when he didn't realise how big RRDtool was going to be!).  Being able to write a struct directly to disk is always going to be more efficient than if you have to swap bytes of integers around first.  And, since (in the current code) we're writing the runtime data structure raw direct to disk, this means that word alignment in structs *is* an issue.

Of course, there are ways around these problems, particularly since the headers structs are infrequently read and so can cope with translation and less efficient writing methods, but this would all require a more significant rewrite of how rrdtool works - and this brings us on to Tobi's RRDTool 2.x project which is going to address this issue, amongst others.

I'm pointing out the issues of why it is difficult to add portability to RRDTool 1.x, not why it is difficult to do at all.  Since you say above "... that is, if it were catered for right at the beginning" I guess you are with me on this one.  RRdtool 1.x did not cater for it at the beginning, hence the problems.  RRDtool 2.x will cater for it from stage one.

In the meantime, the only way to deal with it is to use rrdcached and store the files centrally, with all clients accessing the same files over the network.  This way, the clients can be on different architectures without issue.

Steve

Steve Shipway
University of Auckland ITS
UNIX Systems Design Lead
s.shipway at auckland.ac.nz
Ph: +64 9 373 7599 ext 86487