[rrd-users] HWPredict to filter outlier data on the fly?

[Tobias Oetiker]

HI David,

Yesterday David Purdy wrote:

> Greetings,
>
> Is there a way to use HWPredict to filter errors (ie. outlier
> data) from the data stream, on the fly, at the moment it is being
> collected?

you could use rrdtool updatev to post your updates, then rrdtool
will tell you immediately what it wrote to the RRAs (if it did so).

cheers
tobi
>
> I have a 1-wire temperature sensor that is polled once a minute.  Occasionally a piece of data is obviously a garbage/noise/outlier point, and ideally it should be discarded.  I note that this is a GAUGE situation, but also, (normal) meteorological temperature data will be differentiable with respect to time:  in the (rough) Calculus sense, there can't be any nasty cusps or vertices in the data.
>
> An example showing such a cusp is this ( the 19:24 data is borked) :
>
> <!-- 2014-01-26 19:19:00 CST / 1390785540 --> <row><v>1.2987500000e+01</v>
> <!-- 2014-01-26 19:20:00 CST / 1390785600 --> <row><v>1.2987500000e+01</v>
> <!-- 2014-01-26 19:21:00 CST / 1390785660 --> <row><v>1.2987500000e+01</v>
> <!-- 2014-01-26 19:22:00 CST / 1390785720 --> <row><v>1.2880211103e+01</v>
> <!-- 2014-01-26 19:23:00 CST / 1390785780 --> <row><v>1.2542394264e+01</v>
> <!-- 2014-01-26 19:24:00 CST / 1390785840 --> <row><v>6.7108108375e-01</v>
> <!-- 2014-01-26 19:25:00 CST / 1390785900 --> <row><v>1.0005696817e+01</v>
> <!-- 2014-01-26 19:26:00 CST / 1390785960 --> <row><v>1.2200000000e+01</v>
> <!-- 2014-01-26 19:27:00 CST / 1390786020 --> <row><v>1.1985339718e+01</v>
> <!-- 2014-01-26 19:28:00 CST / 1390786080 --> <row><v>1.1975000000e+01</v>
>
> Graphically, this might look like the spike at about 16:25 hrs :  https://www.dropbox.com/s/zrvi15ez0zbqj2j/1wiretemps_showing_outlierspike.png
>
>
> My current and simplistic solution to this is to discard any data for which the rate of change exceeds a real-world limit.: 3.5 deg (F) per minute.  For instance the fastest recorded temperature drop is 27.2 °C (49 °F) in 15 minutes; Rapid City, South Dakota, 1911-01-10. So, currently I'm just testing the (absolute value of) temperature change over the last minute to see  if it is less than 3.5 degrees. If so, then I'll assume it is good data, otherwise, I'll discard it.
>
> In effect, this requires the absolute value of the 1st derivative to be less than 3.5 (deg F/ min).
>
> Perhaps there is a more intelligent, sophisticated and built-in method for using the Holt-Winters methods (HWPredict), or perhaps the 2nd derivative as well?
>
> If so, could you provide some details on this, perhaps showing syntax and rpn-format?
>
> Thank you,
>
> Dave Purdy
>
>
>
>
> _______________________________________________
> rrd-users mailing list
> rrd-users at lists.oetiker.ch
> https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users
>
>

-- 
Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland
http://it.oetiker.ch tobi at oetiker.ch ++41 62 775 9902 / sb: -9900