[rrd-users] Discrepancy between RRD Average and hand-rolled Average

[Alex van den Bogaerdt]

On Wed, Jun 18, 2008 at 08:10:51AM -0400, Ruttenberg, Tanya wrote:
> I have a script that fetches RRD data for the last 5 days.  I take the
> average of that data (it happens to be inbound utilization) in the usual
> way: sum the data and divide by the number of pieces of data.  The
> result I get is 95097.03.
> 
> I have another script that uses rrdtool graph and VDEF and PRINT to get
> the average of the data from **the same rrdfile**:
> 
> my ($result, $b, $c) = RRDs::graph("/dev/null", "--start", "$starttime",
>    "--end", "$endtime", "DEF:ds0=${file}:$inDS:AVERAGE",
> "VDEF:ds1=ds0,AVERAGE", "PRINT:ds1:%20lf"
> );
> my $inAve1 = $$result[0];
> $inAve1 =~ s/\s//g;
> 
> The result I get from this is quite different than the other one:
> 119465.02
> 
> Any idea why there might be a discrepancy? I use the same start (now - 5
> days) and end time (now) for each

The difference will be in NaN entries.  But first a comment on that
last line of you...

If you use 'now', then chances are *very* slim that you run two
commands with the same time.  You would have to start both programs
*exactly* at the same time, which is hard if not impossible to achieve.


Back to the problem:  NaN entries.  At least that is what I suspect.

the source:
            for (step=0;step<steps;step++) {
                if (finite(data[step*src->ds_cnt])) {
                    sum += data[step*src->ds_cnt];
                    cnt ++;
                };
            }

in english: only add finite entries, and keep count of them.
Later it divides sum by cnt:

            if (dst->vf.op == VDEF_TOTAL) {
                dst->vf.val = sum * src->step;
                dst->vf.when = 0;        /* no time component */
            } else {
                dst->vf.val = sum/cnt;
                dst->vf.when = 0;        /* no time component */
            };



According to what you wrote, you are adding all data (presumably you
alter unknown into zero) and divide by the total amount of time.

These computations

total/time_including_nan =  95097.03
total/time_excluding_nan = 119465.02

We know: time_including_nan = 5 days = 432000 seconds.

We know: total/432000 = 95097.03, therefore total has to be
95,097.03 * 432,000 = 41,081,916,960.00

The real total will be about 50GB in 5 days thus 10GB a day.


Now if 41,081,916,960.00 / time_excluding_nan = 119,465.02, then
time_excluding_nan will have to be 343,882.

We now know:
time_including_nan    = 432000
time_excluding_nan    = 343882
------------------------------
and compute: time_nan =  88118

This means about 100%*88118/432000= 20.4% of your entries are NaN.

Quick check:

VDEF uses 4 out of 5 entries. Your computations use 5 out of 5.
Your calculations will divide by a number 5/4 too high, meaning
your result will be 4/5 of reality. 4/5 * 120,000 is 480,000/5 =
48,000*2 = 96,000.  Close enough to 95097.03; check.


Please let me know if this hunch was correct.

-- 
Alex van den Bogaerdt
http://www.vandenbogaerdt.nl/rrdtool/