[rrd-developers] "transferring data" from an old to a to a new rrd

[Martin Sperl]

Hi!

I have tried to make use of rrdjig.pl, but it ran for ages at 100%CPU and never finished on my test data.

So I started writing a script that is able to extract the data from an original file and produces a list of update lines (takes from the "closest available resolution" - so if we have data for step 300, we take this, if not go to the next-higher version - up to 86400) - the timestamp is EXACTLY at the step boundry.
This output is then fed into a new program that does the updates on an EMPTY rrd file with different RRA definitions (different number of rows and different pdp_per_step).

But when I apply this update to the new empty rrd of the SAME definition i see:
a)  the data at step 300 is identical
b) but the consolidated data at lower resolutions (say 86400) is not identical.

So after heavy debugging I had found out that a lookup of the value in a Specific RRA for a specific Timestamp TS requires the following logic:

For the timestamp TS I want to Insert (assuming TS % <STEP>=0), I have to look up in the value in the RRA at the Timestamp:
LookupTS(TS,RRA) = TS + IF(TS MOD (<STEP>*<RRA.PDP_PER_ROW>)>0,<STEP>*<RRA.PDP_PER_ROW> - TS MOD (<STEP>*<RRA.PDP_PER_ROW>),0)

Maybe someone can shed some light on this mystery, why this logic has to be so complex?

Especially when reading posts on Holt-Winters forecasting it always says "gather data, then reimport them into an rrd definition that has been created with different parameters for Holt-Winters" - I assume most people made similar assumptions on the timestamps when extracting the data via RRDDump or RRDFetch.

Thanks,
		Martin

P.s: If there is interest, then I will share the script as soon as It handles also counters and not only GAUGES (and probably ABSOLUTE - an assumption)... If I get it right it should also be able to handle multiple source rrd files and allow joining them to a single RRD. In the future the script potentially can do the updates itself as well (without the pipeing)...