[rrd-users] how change pdp_per_row ??

[Alex van den Bogaerdt]

----- Original Message ----- 
From: "petterborgca" <petterborgca at ldschurch.org>
To: <rrd-users at lists.oetiker.ch>
Sent: Monday, October 27, 2008 7:28 PM
Subject: Re: [rrd-users] how change pdp_per_row ??


>
> I was looking through the archive in the forum for an answer to my 
> problem,
> and I found this message, but there was no real answer.
>
> Here is my problem. We want to increase the number of rows for a greater
> length of time (which I can do with resize) AND the pdp for more precision
> (which is where I'm having my issue).

You may want to elaborate a bit on 'increasing the pdp'.


> I thought I might be able to do it by doing a resize, then dump, then edit
> the pdp_per_row value in the XML and then doing a restore. Ahh... no. 
> There
> is a timestamp for each row, but it is just a comment, so the data is now
> skewed to a different time frame.
>
> I *could* create a new rrd file with the RRAs set up the way that I want,
> then do a series of updates (one for each row in the old rrd), but that
> might not get me what I want either. We want our long term RRD to change
> from 240 pdp to 12 pdp and from 480 data points to 17,532 points.

Are you saying here that you have just one RRA, with each row being built 
from 240 PDPs, and you want to change this to one RRA with each row being 
built from 12 PDPs ?

> Any suggestions for how to convert an rrd file to hold more data and more
> frequent data points without messing up the data? Yes, I realize that I
> can't have it make up the data points that are missing. I'm not even sure
> this would lead to usable data in the file, but that is what we would like
> to do.

Basically what you have available is rates, thus type GAUGE. And these do 
come with timestamps.

Two possible approaches to this problem. One uses 'rrdtool fetch' a couple 
of times. Having more than one consolidation type (AVERAGE, MIN, MAX, LAST) 
will be a huge challenge to get it right. Originally I wrote 'is 
impossible', but that wouldn't be true.

1: Create a new database, with a start time just before the beginning of 
your data series
2: Alter its properties (RTFM rrdtool tune) so that:
2a: it accepts updates that are otherwise too far apart (RTFM heartbeat)
2b: it takes rates (GAUGE data type)
3: update the new database using data from the old database
4: Alter the new database's properties back to what you want them to be

The other would be using rrdtool dump and restore:

1: dump the database
2: alter the xml file. You will want to alter 'step' (which is the amount of 
time per PDP) and you will want to alter the data (as this is based on step) 
or the RRA properties.  For instance: if you decrease the amount of time per 
PDP, you can double the amount of rows in one RRA and double the amount of 
PDPs per CDP in another RRA.  When adding more rows, you will need to insert 
those with a proper timestamp.
3: restore the database

Say you double the amount of rows and at the same time divide the amount of 
time in each row by 2.

The next example would work if:

a) step==10 and steps==6  =>  step==10 and steps==3
b) step==10 and steps==6  =>  step==5 and steps==6
c) step==60 and steps==1  =>  step==30 and steps==1
d) step==20 and steps==3  =>  step==5 and steps==6
and similar. In this example I'm changing 60 seconds per entry to 30 seconds 
per entry.


Say you start with this:
<!-- 2008-06-01 11:10:00 CEST / 1212311400 --> <row><v> 7.0921985816e-02 
</v><v> 0.0000000000e+00 </v><v> 6.7375886525e-04 </v><v> 7.0921985816e-04 
</v></row>
<!-- 2008-06-01 11:11:00 CEST / 1212311460 --> <row><v> 1.1111111111e-01 
</v><v> 0.0000000000e+00 </v><v> 2.9314420804e-03 </v><v> 7.8014184397e-04 
</v></row>

The rates valid between 1212311400 and 1212311460 are also valid between 
1212311400 and 1212311430, and they are also valid between 1212311430 and 
1212311460.

That means I can take the 2nd line, make a copy, place it inbetween and 
alter its timestamp like so:

<!-- 2008-06-01 11:10:00 CEST / 1212311400 --> <row><v> 7.0921985816e-02 
</v><v> 0.0000000000e+00 </v><v> 6.7375886525e-04 </v><v> 7.0921985816e-04 
</v></row>
<!-- 2008-06-01 11:10:30 CEST / 1212311430 --> <row><v> 1.1111111111e-01 
</v><v> 0.0000000000e+00 </v><v> 2.9314420804e-03 </v><v> 7.8014184397e-04 
</v></row>
<!-- 2008-06-01 11:11:00 CEST / 1212311460 --> <row><v> 1.1111111111e-01 
</v><v> 0.0000000000e+00 </v><v> 2.9314420804e-03 </v><v> 7.8014184397e-04 
</v></row>

Effectively this should look to rrdtool as if you updated a database at time 
1212311400 and at time 1212311460, with a heartbeat of at least 60 seconds. 
It looks as if two CDPs were filled with just one update command.  As long 
as I make sure { 'step' (amount of time per PDP) times 'pdp_per_row' aka 
'steps' (amount of PDP per CDP) times the amount of rows } stays the same, I 
didn't hurt my data.

I could also insert two extra lines and change step and/or steps so that 
they multiply to 20. As long as you understand that '1212311460' really 
means '1212311460-something until 1212311460', and you understand what this 
'something' is, you're fine.


Please be aware:

1: you are messing with the data. This could go wrong. Make sure you can 
read and use your backup files. If possible you also would want to 
(temporarily) save your update commands so that you can restore the previous 
database and redo the updates.
2: whenever messing with the database, you will want to do so when you've 
updated it at precisely the right time. This is when the data has been 
written to the RRA *and* no data is left to process, meaning the most recent 
line in a dump has a timestamp which is exactly the same as you've used in 
your update command. This is so that you don't have to worry about the 
cdp_prep part.
3: data in a database is not just a rate, it is a combination of rate and 
duration (implied).
4: you may loose some precision when doing stuff to the data. 
1.1111111111e-01 may or may not be the same as 
1.111111111111111111111111111111111111111111111e-01. This said, differences 
will be small and hardly noticable if at all.


It took me half an hour to write this message, and it may still not be 
enough for you to understand.  If that is the case, you may want to decide 
to not DiY but to hire someone who is capable of doing this.  I'm available 
but feel free to shop around.

regards,
Alex