[rrd-users] Calculating AVG, treating UNKNOWNs as zero

[Derek Haynes]

Alex,

> And is that a big problem?  I never measured it, I would guess there is a
> little performance penalty because rrdtool will output an error message for
> you to ignore, but maybe I'm wrong and maybe that does take a lot of
> resources (on a large scale setup).

Yeah - for us, the best behavior would be for RRDTool to ignore a
duplicate time when performing a mass update and proceed with the
later valid updates. However, once the illegal update is encountered,
further updates are ignored. This means the error message can't be
ignored.

> Will it happen often that there was an update in the previous minute?

There's a wide variation.

What I'm leaning towards:

* Adding a long heartbeat (1 week)
* Caching the last update time
* Before updating, if the last update is > 1 minute ago, insert zero 1
minute prior.

-Derek

On Thu, Aug 2, 2012 at 8:13 AM, Alex van den Bogaerdt
<alex at vandenbogaerdt.nl> wrote:
>
> ----- Original Message -----
> From: "Derek Haynes" <derek.haynes at highgroove.com>
> To: "Alex van den Bogaerdt" <alex at vandenbogaerdt.nl>
> Cc: <rrd-users at lists.oetiker.ch>
> Sent: Wednesday, August 01, 2012 10:54 PM
> Subject: Re: [rrd-users] Calculating AVG, treating UNKNOWNs as zero
>
>
>> Hi Simon/Alex,
>>
>>> use a CDEF like b=a,UNKN,0,a,IF
>>
>> I don't believe this works as I'm fetching from a larger step size (30
>> minutes) than the RRA (1-minute) and the CDEF performs the RPN logic
>> on the DEF at the 30-minute step. At this point, the average already
>> is ignoring the unknowns.
>
> That is also what I thought, but I also think this is why you want equal
> sized buckets of data, so updating at time t-1 is wrong (for you) regardless
> of which value you put in.  When averaging, you want the 12 in {10,11,12} to
> be of the same weight as the 12 in {0,0,12}.
>
>>> Details may vary, depending on the information you left out, but IMHO
>>> this
>>> is the path to your solution
>>
>> Thanks - based on the high-level summary I outlined, your solution
>> works. However, there's a performance reason why it won't for our
>> setup:
>>
>> * Before updating a file, I don't know when the last update occured.
>
> But do you at least know it will have happened a whole number of minutes
> ago?
>
>> We're updating many files at once - for performance, we want to avoid
>> reading the lastupdate of each RRD file before updating to determine
>> if we need to proceed the update with time-60:0. If a file received an
>> update in the previous minute, updating again with zero will result in
>> an error.
>
> And is that a big problem?  I never measured it, I would guess there is a
> little performance penalty because rrdtool will output an error message for
> you to ignore, but maybe I'm wrong and maybe that does take a lot of
> resources (on a large scale setup).
>
>> * At most, we update these files once per-minute. Updating w/an
>> UNKNOWN one-second prior is always safe.
>
> But at the same time it is always what you not want. You do not want to
> update at T-1, and you want to avoid unknowns.  I really believe you should
> step back and look at your real problem. What you are trying to do now is to
> solve another problem, which you introduced yourself as a result of a wrong
> fix. Forget about the unknowns; inserting them is not the solution to your
> problem. Do not insert them, and you do not need to fix this problem.
>
> Will it happen often that there was an update in the previous minute? If
> not, you probably can ignore the extra cycles needed to output an error
> message.  Just try updating the previous minute with zero. If that fails a
> few times, you probably not need to care. If it does happen a lot, maybe you
> need to find a way for your application to remember if it updated the
> database or not.
>
> HTH
> Alex
>
> _______________________________________________
> rrd-users mailing list
> rrd-users at lists.oetiker.ch
> https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users



-- 
Derek Haynes
Scout Web Monitoring and Reporting ~ http://scoutapp.com
Blog ~ http://blog.scoutapp.com
415.871.7979