[rrd-users] Re: Using RRD for sparse and generic statistical data

[Alex van den Bogaerdt]

Mike Papper wrote:

> Hello, I have just started investigating RRD as a tool for counting
> statistics.

Welcome !

> These statistics are numbers that represent the occurance of an event at a
> particular time
> (or within a given restricted time period). Thus we store things like
> "user logins occured 10 times this past 5 minutes' or
> "user logins occured 0 times this past 5 minutes"
> "user logins failed 0 times this past 5 minutes" etc.

This is what RRDtool is designed for.

> The set of stats are ever increasing - there are new stats added every
> time a programmer programs a new function.
> So if the system now supports "proxy logins" then we would add a new
> stat to represent this. The stats is added "on the fly" and the
> corresponding RRD file is setup (via a script) on-the-fly. Thus when a
> programmer adds a new stat, a new RRD is created for it automagically.

Perfect.  This is the way to go, as you don't really have an option to
add more DSes to an existing RRD.

[snip]

> There will be many many generated stats. The detailedness (resolution down
> to an hour) should be approx. the same for all stats.

> Question 1: For many of the RRDs, the "step" will not be adhered to.
> If I set the step to 30 minutes, I may "update" date into an RRD
> only every 30 mins or once a day. What I want is for RRD to take
> the case of "no data at a specified time interval" to mean 0 not unknown (I
> think).

Not really a question here.  I'll assume that you want to know if this
is the way to do it.  I don't think so.  If you know there was nothing
in a certain interval, just let RRDtool know this.  To quote you again:
> "user logins occured 0 times this past 5 minutes"
This means that you know there were zero logins.  Tell RRDtool and you
will be much happier.  You can setup a heartbeat timer in your front end,
if no information is sent to RRDtool for (for instance) half an hour the
front end feeds the number 0 to RRDtool.

> For example: a login fails 2 times in the past 30 minutes. We send a
> timestamp and the value "2" to RRD.  Then no more login failures occur
> for a day - so we dont send any data to the RRD.  Then the next day 1
> failure occurs so we send the timestamp and a value of 1.

Same as my previous paragraph.

Also make sure to understand "step" and "heartbeat" (and maybe "xff").
Step defines the interval in the RRAs, hearbeat defines when an update
causes unknown values.


> Question 2: I have many servers running. Each on its own machine.
> Each can generate 'stats' that are the same name (and the same period).
> For example, there are 3 login servers and anyone can generate a
> "login.failure" stat at the same time. And each will send its data to
> an RRD independently.  Thus the RRD may receive 3 sets of data for the
> same RRD and the same time period.
> 
> Is this OK? I.e., will RRd accumulate values for the same time period
> or will it overwrite them?
> 
> How can I make it accumulate values (withing RRD)?

You need a front end.  It doesn't matter if you use an existing one or
you write your own, you need one.  Even a small script that does nothing
more than entering a certain variable into an RRD can be called a front
end.

It is the responsibility of the front end to collect and process the
values.  Three logins failing during the same second means that the
front end should feed the number three to the RRD.

RRDtool can handle updates one second apart.  It cannot cope with two
updates occuring at the same time (with a one-second resolution) and
it cannot go back in time.

If you have to deal with delayed data (such as "login failed" msgs
coming in only after a few seconds) the front end needs to accumulate
this information and feed it to RRDtool at a suitable time.  You can
tell RRDtool that this event happened at a certain time, as long as
no other update has occured after that time (or: at the same time).

HTH,
-- 
   __________________________________________________________________
 / alex at slot.hollandcasino.nl                  alex at ergens.op.het.net \
| work                                                         private |
| My employer is capable of speaking therefore I speak only for myself |
+----------------------------------------------------------------------+
| Technical questions sent directly to me will be nuked. Use the list. | 
+----------------------------------------------------------------------+
| http://faq.mrtg.org/                                                 |
| http://rrdtool.eu.org  --> tutorial                                  |
+----------------------------------------------------------------------+

--
Unsubscribe mailto:rrd-users-request at list.ee.ethz.ch?subject=unsubscribe
Help        mailto:rrd-users-request at list.ee.ethz.ch?subject=help
Archive     http://www.ee.ethz.ch/~slist/rrd-users
WebAdmin    http://www.ee.ethz.ch/~slist/lsg2.cgi