<div dir="ltr">Note that variance, and hence stddev, can be calculated incrementally (by keeping a timeseries of the average rate squared; variance = (average rate^2 - average^2), stddev=sqrt(variance)), and assuming a normal distribution, 95th percentile = 2*stddev. The accuracy of this depends on how closely your samples match a normal distribution and is not as resilient to outliers as calculating a true 95th percentile from all the samples, but it's a pretty good approximation. If you know your distribution is closer to log-normal (which it often is for things like latency), you can calculate a more accurate 95th percentile from the average and variance like this;<div><br></div><div><div>mu = ln(avg) - ln(var/avg**2 + 1)/2</div><div>sigma = sqrt(ln(var/avg**2 + 1))</div></div><div>p95 = lognorminv(0.95, mu, sigma)<br></div><div><br></div><div>Unfortunately right now rrd doesn't support RRA's of type variance (CF=VAR?) or mean value squared (CF=AVERAGE2?). However, if you were going to request a feature, this is something that is definitely possible. A true 95th percentile RRA is definitely not. Another ugly approximation uses bucketed distributions, but I wouldn't request that.</div><div><br></div><div>Note having an RRA of type CF=AVERAGE2 is useful for calculating the "root mean square", something that is also useful for eg AC power calculations. Also, stddev is actually the "root mean square" of the distance from the mean.</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On 25 October 2015 at 10:43, Steve Shipway <span dir="ltr"><<a href="mailto:s.shipway@auckland.ac.nz" target="_blank">s.shipway@auckland.ac.nz</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div style="direction:ltr;font-family:Tahoma;color:#000000;font-size:10pt">Percentiles cannot be calculated incrementally; you need the entire dataset to deduce them whereas mean, max, min only require the last calculation result and possibly the number
of samples so far. Hence you cannot have the percentile as a CF<br>
<br>
However the RRDTool RPN functions include a percentile calculator, so you can still deduce this on the fly as you graph using the available samples. You would need to be careful to ensure that the data series over which you are aggregating is of maximum granularity
though if you want to ensure maximum accuracy<br>
<br>
Steve<br>
<div><br>
<div style="font-family:Tahoma;font-size:13px">
<div style="font-family:Tahoma;font-size:13px"><strong>Steve Shipway</strong></div>
<div style="font-family:Tahoma;font-size:13px">University of Auckland ITS</div>
<div style="font-family:Tahoma;font-size:13px"><em>UNIX Systems Design Lead</em></div>
<div style="font-family:Tahoma;font-size:13px"><a href="mailto:s.shipway@auckland.ac.nz" target="_blank">s.shipway@auckland.ac.nz</a></div>
<div style="font-family:Tahoma;font-size:13px">Ph: <a href="tel:%2B64%209%20373%207599%20ext%2086487" value="+6493737599" target="_blank">+64 9 373 7599 ext 86487</a></div>
<div style="font-family:Tahoma;font-size:13px"><em></em> </div>
</div>
</div>
<div style="font-family:Times New Roman;color:#000000;font-size:16px">
<hr>
<div style="direction:ltr"><font face="Tahoma" color="#000000" size="2"><b>From:</b> rrd-users [rrd-users-bounces+s.shipway=<a href="mailto:auckland.ac.nz@lists.oetiker.ch" target="_blank">auckland.ac.nz@lists.oetiker.ch</a>] on behalf of Pablo Chacin [<a href="mailto:pchacin@sensefields.com" target="_blank">pchacin@sensefields.com</a>]<br>
<b>Sent:</b> Saturday, 24 October 2015 11:43 p.m.<br>
<b>To:</b> <a href="mailto:rrd-users@lists.oetiker.ch" target="_blank">rrd-users@lists.oetiker.ch</a><br>
<b>Subject:</b> [rrd-users] Percentile consolidation<br>
</font><br>
</div><div><div class="h5">
<div></div>
<div>
<div dir="ltr">
<div>
<div>Greetings</div>
<div><br>
</div>
<div>Been able to pre-calculate an store certain data percentiles, like media an 95 percentile is a common requirement for any metrics database, as these aggregation functions are much more stable and representative of data than the average or maximun values.</div>
<div><br>
</div>
<div>I saw that the mean was recently included as an consolidation function in rrdtool, but still there's no possible to calculate other arbitrary percentiles. Interestingly, percentiles have been available when retrieving data for graphs or reporting.</div>
</div>
<div><br>
</div>
<div>Is there any compiling reason not to include percentiles as consolidation functions? Is there any plan to do so in the future?</div>
<div><br>
</div>
<div>Regards</div>
<div><br>
</div>
<br clear="all">
<div>
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div style="font-family:arial;font-size:small">---------------------------</div>
<div style="font-family:arial;font-size:small">Pablo Chacin</div>
<div style="font-family:arial;font-size:small">CTO</div>
<div style="font-family:arial;font-size:small">SenseFields SL</div>
<div style="font-family:arial;font-size:small">Tlf (+34) 93 <a value="+34932504598" style="color:rgb(17,85,204)">250 45 98</a></div>
<div style="font-family:arial;font-size:small">Gran Via 674, principal 1º</div>
<div style="font-family:arial;font-size:small">08010 Barcelona, Spain</div>
<div style="font-family:arial;font-size:small"><a href="http://www.sensefields.com/" style="color:rgb(17,85,204)" target="_blank">http://www.sensefields.com</a> </div>
<div style="font-family:arial;font-size:small"><br>
</div>
<div>
<div><br>
</div>
<div><font face="arial narrow, sans-serif" size="1">This message was directed exclusively at the recipient and contains privileged and confidential information. If you receive this message in error, I beg to inform us immediately by reply email or by phone
0034 93 250 45 98, and proceed to their elimination.</font></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div></div></div>
</div>
</div>
<br>_______________________________________________<br>
rrd-users mailing list<br>
<a href="mailto:rrd-users@lists.oetiker.ch">rrd-users@lists.oetiker.ch</a><br>
<a href="https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users" rel="noreferrer" target="_blank">https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature">Donovan Baarda <<a href="mailto:abo@minkirri.apana.org.au" target="_blank">abo@minkirri.apana.org.au</a>></div>
</div>