[rrd-users] Calculating AVG, treating UNKNOWNs as zero

[Derek Haynes]

Hi Simon/Alex,

> use a CDEF like b=a,UNKN,0,a,IF

I don't believe this works as I'm fetching from a larger step size (30
minutes) than the RRA (1-minute) and the CDEF performs the RPN logic
on the DEF at the 30-minute step. At this point, the average already
is ignoring the unknowns.

> Details may vary, depending on the information you left out, but IMHO this
> is the path to your solution

Thanks - based on the high-level summary I outlined, your solution
works. However, there's a performance reason why it won't for our
setup:

* Before updating a file, I don't know when the last update occured.
We're updating many files at once - for performance, we want to avoid
reading the lastupdate of each RRD file before updating to determine
if we need to proceed the update with time-60:0. If a file received an
update in the previous minute, updating again with zero will result in
an error.
* At most, we update these files once per-minute. Updating w/an
UNKNOWN one-second prior is always safe.

Any thoughts?

Derek

On Tue, Jul 31, 2012 at 3:21 AM, Alex van den Bogaerdt
<alex at vandenbogaerdt.nl> wrote:
>
> ----- Original Message -----
> From: "Derek Haynes" <derek.haynes at gmail.com>
> To: <rrd-users at lists.oetiker.ch>
> Sent: Monday, July 30, 2012 8:54 PM
> Subject: [rrd-users] Calculating AVG, treating UNKNOWNs as zero
>
>
>> The AVG consolidation function excludes UNKNOWN values. Instead of
>> excluding them, I'd like to convert each UNKNOWN value to zero and
>> then generate the average.
>
> In your example, you deliberately update with "Unknown", and then you want
> to get rid of it.
> Why?
>
>> Example use case: recording visits to a URL, updating once-per minute
>> (but only if any visits), and then calculating the throughput over a
>> given period.
>
> So what you know is: the amount of visits in a minute, which may be zero.
> Then store that, and I think your problem is gone.
>
>> Example ----
>>
>> 1-minute Average RRA. 1 second step:
>>
>> rrdtool create test.rrd \
>> --start 1325376000 \
>> --step 1 \
>> DS:x:GAUGE:60:U:U \
>> RRA:AVERAGE:0.99:60:10080
>>
>> Updates - 1 at Jan 1 00:01, another at Jan 1 00:30:
>>
>> rrdtool update test.rrd \
>> 1325376060:100 \
>> 1325377799:U \
>> 1325377800:25
>>
>> ...this means the values for the CDPs from 00:02 - 00:29 will be
>> UNKNOWNs. The 2nd update above forces the 3rd to be recorded.
>
> The problem you have: that 25 is not going in the database because it is
> ignored because it is too late.
> You tried to solve this problem by updating with unknown, a second before
> the actual update you're going to perform.
> Additionally, you want to have those 25 be valid from
> 1325377740..1325377800, just like the first update should be valid from
> 1325376000..1325376060 (correct?)
>
> In this example you know that between 1325376060 and 1325377740 there were
> no visits, so I would solve this by increasing the heartbeat to a large
> number (could be as large as a week even) and updating at time 1325377740,
> not 1325377799, with zero, not with unknown.
>
> This would result in uniform buckets of 60 seconds, with the rate you set or
> with a rate of zero.
>
> Details may vary, depending on the information you left out, but IMHO this
> is the path to your solution
>
> _______________________________________________
> rrd-users mailing list
> rrd-users at lists.oetiker.ch
> https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users



-- 
Derek Haynes
Scout Web Monitoring and Reporting ~ http://scoutapp.com
Blog ~ http://blog.scoutapp.com
415.871.7979