[rrd-users] convert rrd into new rrd with different RRAs

[rrd-users at ficos.de]

Tobias Oetiker schrieb:
> Hi Stefan,
> 
> there is no such program to that does this as fahr as I can tell
> ... the simples way to achieve this is to write an app that pulls
> the data from the original rrd, sorting the updates by time picking
> the higest resolution available for each instant and then feeding
> it back to a newly setup rrd ... it will get trickier if you want
> to support MAX RRAs as well, but you get the idea ?
> 
> cheers
> tobi

I've tried this aproach before, but with large rrds this takes forever,
because it always feeds the finest resolution first and then runs through
all aggregation steps. I have rrds with values for years, so they're really
large ...

Now lets say I'm sorting the updates by time picking the higest resolution
available, that highest resolution might be 10m (after 2 weeks in that given
example), but I'm feeding those 10m values into the new rrd which has a 2s 
step size, so it has to go through all the aggregation steps for all DPs.

A smarter solution might be: create multiple new rrds, one for each wanted
rra-size, and then feed each of the new rrds with the resolution suitable for 
it's rra-size, eg. not the highest, but the lowest that is still higher than
the new wanted.
After having done that, combine the multiple rrds into one rrd with multiple
rras. That is much faster, but much more complicated :-(

It get's even more trickier if you need to keep track on 'gaps' in the data,
because the system was offline for a while. The gap might be large enough to
trigger the heartbeat/xff for the first rra, but not for the others ...
That makes it even more difficult ...

I have postponed this project to some long dark cold winter nights ... :-/

- Karl






 
> Today Stefan M. Brandl wrote:
> 
>> On Mon, Mar 23, 2009 at 07:32:07PM +0100, Karl Fischer wrote:
>>> Hi,
>>>
>> Hello Karl,
>>
>>> I'm re-posting this since I didn't get a single reply - any ideas?
>>>
>>> I'm trying to convert many existing rrds into new rrds with different RRAs.
>>>
>>> so for example I might have rrd's created with:
>>> (RRAs: 1s, 10s, 60s, 600s)
>>>
>>>   --step=1
>>>   DS:somevalue:GAUGE:120:0:U
>>>   RRA:AVERAGE:0.5:1:3600    # 1s for 1h
>>>   RRA:AVERAGE:0.5:10:8640   # 10s for 1d
>>>   RRA:AVERAGE:0.5:60:10080  # 60s for 1w
>>>   RRA:AVERAGE:0.5:600:4320  # 10m for 1m
>>>
>>> and I'd like to convert this into a new rdd that looks like this:
>>> (RRAs: 2s, 10s, 60s, 900s)
>>>
>>>   --step=2
>>>   DS:somevalue:GAUGE:60:0:U
>>>   RRA:AVERAGE:0.5:1:5400    # 2s for 3h
>>>   RRA:AVERAGE:0.5:10:8640   # 10s for 1d
>>>   RRA:AVERAGE:0.5:30:20160  # 60s for 2w
>>>   RRA:AVERAGE:0.5:300:12960 # 10m for 3m
>>>
>>> Of course that means that some data needs to be calculated from higher
>>> RRAs, for example:
>>>
>>> the first half of the 2s RRA in the new database can be taken 1:1 from the
>>> 1s RRA of the exiting database by using the CF on two datapoints each time.
>>> The second half is more tricky since there is no data in 1s resolution in
>>> the first RRD, so the values need to be interpolated from the 10s RRA ...
>>>
>>> I wonder if I have to re-invent the wheel and to write the required code
>>> completely myself or if someone has been there already ... ?
>>>
>>> Any hints welcome ...
>>>
>> As I did not see any hints to the mailing list, I wanted to
>> ask if there are any news on this topic?
>>
>>
>> Stefan