[rrd-users] How to graph slow changing counters

[Simon Hobson]

Thorsten Erdmann <thorsten at trektech.de> wrote:

> you are right (and I know) that with such a low resolution it is difficult to draw sensefull data. But I want to understand what RRD does. I think the aggregation is wrong:
> 
> If I sample the data every 5 minutes (1/12h) and get a single 1kWh step in one of these 5 minute intervals then I get a peak about 0,3kW in the dayly graph. But why? I get headache with these calculation. I would say the formular is 1kWh / (1/12)h but that gives 12kW, which is definitely wrong.
> 
> Another weird thing is what happens on the weekly graph. I get some "nice" step lines up and down. Shouldn't it be simply a flattened out curve just as you said. I cannot give you a picture right now, because my server is down for implementing the S0 photosensor, just as you suggested. I will implent both. So I can adjust the S0 pulse counter once a day or so to the real counter value.



> This is how I defined the data:
>  
>                 ret = rrdtool.create(rrdname + '.rrd', '--step', '300', '--start', str(starttime),
>                     'DS:Gauge:GAUGE:600:U:U',
>                     'DS:Counter:COUNTER:600:U:U',
>                     'RRA:AVERAGE:0.5:5m:24h',
>                     'RRA:AVERAGE:0.5:2h:31d',
>                     'RRA:AVERAGE:0.5:6h:31d',
>                     'RRA:AVERAGE:0.5:1d:1y')
>  
> And this is how I graph them
>  
>     ret = rrdtool.graph(rrdname + '24h.png', '--start', 'end-24h', '--end', 'now', 
>         '--upper-limit', '650', '--lower-limit', '600', '--rigid', '--right-axis', '0.00001:-0.006',
>         'DEF:gauge=./' + rrdname + '.rrd:Gauge:AVERAGE',
>         'DEF:counter=./' + rrdname + '.rrd:Counter:AVERAGE',
>         'CDEF:cnt1000=600,counter,10000,*,+',
>         'LINE1:gauge#00ff00:Gauge',
>         'LINE1:cnt1000#ff0000:Counter')
>  
> The gauge line looks good, but the counter line looks weird.


Looking again I have some questions and observations ...

You have a GAUGE and a COUNTER in your rrd - what are you collecting, and how ? Counter would be correct for the meter, what are you putting in the gauge ?
Knowing this may affect the advice offered.

I'd recommend you store a bit more data than you are (e.g. 25h for daily graphs) - and that links in with graphing.
As you've written them here, I think you'll find the wrong aggregate data being used. If you do "now-24h" to "now" then you'll find that the first data point in that period is just outside your 24 hours @ 5 minute - so instead you'll find that the 2 hour aggregation is used (see more below). For that reason I've always calculated the exact end time of the last period - i.e. in bash, something like :

step=300
durn=87600
etime=`date +%s`
etime=$((${etime}/${step}*${step}))
stime=$((${etime}-${durn}))
... rrdtool ... "--start ${stime} --end ${etime} --step ${step} ...

That way, your strt and end times are aligned to the bucket boundaries of the RRD file, and you've explicitly defined the data resolution to be used. This gives more reliable selection of data !
I tend to set step and durn in a case statement, along the lines of :
case period in
  daily ( step=300;durn=87600 );;
  weekly ( step-7200; ...
and then the graphing command is common to all resolutions.


A bit more on how RRDtool selects data. Bear in mind this is from memory which may have faded a bit ...
If you don't tell it otherwise, it will select the highest resolution data that supplies the entire graphing period selected. So selecting "now" as the end point, you will be selecting an incomplete bucket at the end of the graph. From memory, I think this also means that "now - 24h" goes part of a bucket past the first bucket in the RRA - because when the current incomplete bucket is started, it takes the place of the one from 24h ago.
As this goes beyond the 300s stored data, the next resolution is selected - hence you day graph that should show 300s buckets, is actually drawn with 2hr buckets - so coarser data AND potentially a gap at the end (e.g. if "now" is 11:55 UTC, you'll only see data up to 10:00 UTC because the 10:00 to 12:00 UTC bucket is not yet complete).


Simon