[rrd-users] False positives with aberrant behavior detection

[Dave Plonka]

Hi Mike,

On Tue, Aug 17, 2010 at 06:39:27PM -0700, Mike Schilli wrote:
> On Mon, 16 Aug 2010, Tobias Oetiker wrote:
> 
> > you may find some inspiration here
> > svn://svn.oetiker.ch/rrdtool/trunk/tutorial/lisa2009/rrd-by-example
> 
> What would be helpful is a collection of use cases, and explanations on
> how to remedy common problems with either too many false positives or
> obvious errors that aren't picked up.
> 
> "Twiddle the 4 knobs until it looks right" is neither a valued
> engineering tradition (at least where I come from) nor a guarantee that
> it won't be out-of-whack again tomorrow.
> 
> Curious: Since you mention in one of the .tex files that "no one
> considers himself clever enough to use it": Are there any applications
> in the wild that have been using the feature for a while with satisfying
> results? I'd be interested if someone can actually come up with a set of
> parameters so that the anomaly in [1] gets detected without false
> positives. It's an easy case and should be fast to solve.

What's an easy case?  Human beings can't detect problems w/o false
positives so I wouln't expect that of this technique either, but
certainly they can be reduced.  Without careful tuning to the data at
hand, you generally can only eliminate false positives by introducing
false negatives.  What I would suggest, instead (as was, I believe,
Brutlag's original intent) is to be able to alert operators (and
direct their attention) to likely anomalies faster than one could
find them manually solely by visual inspection visually.

> As it stands, and with the confusing results I'm getting, I'm wondering
> if rrdtool's implementation of Holt-Winters is useful at all, or even
> correct.

It does seem that the current implementation differs from what I
was using in rrdtool c. 2002.  My recollection was that there was
the ability to widen the confidence band by a parameter and also to
have different settings for the upper vs. lower bound offset from
the forecast value.

BTW, there is another implementation described here:
   http://www-iepm.slac.stanford.edu/monitoring/forecast/hw.html

Dave

-- 
plonka at cs.wisc.edu  http://net.doit.wisc.edu/~plonka/  Madison, WI