[rrd-users] Percentile consolidation

[Donovan Baarda]

On 27 October 2015 at 03:59, Pablo Chacin <pchacin at sensefields.com> wrote:
>
> From the answers by Steve and Donovan, it seams like there's something
fundamental I'm not understanding about RRDTools.
> Following is the explanation from Wikipedia
[...]
> My understanding was that when making the consolidation, al the primary
data points were available to the Consolidation function. Therefore, it
could be possible to calculate the percentile.
>
> However, from what you explain, it looks as RRDTools re-calculates the
aggretate point with each arriving primary point. Is that correct?

Correct. The aggregation of PDP's into CDP's is done incrementally at each
input sample.

Note that even if this was not the case, calculating meaningful percentile
values would require large "steps" values on the corresponding RRA's. To
calculate a reasonable 95th percentile requires at least 20 points.

This thread made me look at the RRD docs again and I discovered it now
supports DS's of type COMPUTE. That means it can already support
calculating value^2 as another DS, making it possible to do this;

rrdtool create traffic.rrd \
   --start now --step 1m \
   DS:rate:COUNTER:2m:0:1000000 \
   DS:rate2:COMPUTE:rate,rate,*
   RRA:AVERAGE:0.5:1m:8d \
   RRA:AVERAGE:0.5:1h:64d \
   RRA:AVERAGE:0.5:1d:2y \

and then get an approximate 95 percentile in your graphs by calculating
2*stddev like this;

DEF:rate=/home/rrdtool/data/traffic.rrd:rate:AVERAGE
DEF:rate2=/home/rrdtool/data/traffic.rrd:rate2:AVERAGE
CDEF:variance=rate2,rate,rate,*,-
CDEF:stddev=variance,SQRT
CDEF:95ptile=stddev,2.0,*

Unfortunately it doesn't look possible to calculate an approx 95 percentile
assuming a log-normal distribution, because RRD doesn't seem to have the
functions necessary to calculate lognorminv(), at least not easily.

--
Donovan Baarda <abo at minkirri.apana.org.au>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.oetiker.ch/pipermail/rrd-users/attachments/20151027/1330b84f/attachment-0001.html>