[rrd-users] VISIONARY: New tool to consider to create: a "Dynamic Data Set" tool...

[Jakob Ilves]

Hello!

(This is long.  My apologizes for not being able to express what I need
to say with fewer words.  Still, I think it is worth reading).

We are all familiar with the tools like MRTG and RRDtool, which gathers
statistics and presents that in clear graphics (and allows for other
presentations as well, I know).

The statistics these tools works with are statistics for... (how should
I put this consisely...) "invariant sets of variables".  When I say,
"invariant set", I mean that once you configured MRTG/RRDtool to collect
data for a number of targets, resulting in a number of variables, it
then continues to collect and display ONLY THAT SET of variables.  The
set of variables doesn't change over time (even if their actual values
do), unless you explicitly reconfigures it by adding or removing
targets.

This works great in a lot of situations and for a lot of data sources.
The utilization of an interface for example is one single variable.  The
variable itself might vary wildly in value, but it still is that single
variable you want to monitor and graph.

There are situations when this will not work.  Assume you have a method
of  getting reliable statistics for the destination hosts of incoming
traffic on a WAN link and that these stats gives for every five minutes
which host traffic (by IP-address) was recieving data to and how much.
Assume you want to store this information and to do some presentation of
it.

(How to get that information?  Well, RMON2 probes could provide that if
you write code to extract that data out of them.  Cisco routers using
the Netflow feature combined with Netflow exports could be another
way... but that's off topic)

Why will not RRDtool do here?  Well, during one 5 minute period, there
might be, say 30 hosts showing up as destinations, each of them
recieving a certain amount of data.  The next 5 minute period, you don't
know how many of these hosts will show up again (if any).  Also, you
don't know how many new hosts there will be that shows up.  The 5
minutes after that, again, ANOTHER set of hosts appears, each with it's
individual contribution of data it recieved on the WAN link.  So for
each 5 minutes sample, you don't know which hosts (variables) will
appear.  For each sample period, the SET OF VARIABLE MIGHT CHANGE.
RRDtool doesn't support that, at least not in the current version
(assumed by me to be 1.0.28).

(I'm assuming above that the number of hosts being able to recieve
traffic is "large", i.e it's not possible to create a single MRTG target
to take care of the traffic being sent to each host because it would
create a "large" number of MRTG targets).

So what am I suggesting?  It's very short to express but probably not
trivial to achieve: write a tool which efficiently and accurately enough
might store, process and graph this kind of dynamicly changing data sets
and which supports the data consolidation over time MRTG/RRDtool uses
(store 5 min samples for 48 hours, store 30 min samples for 2 weeks
etc)..

There is quite a few things which will be worth to discuss:

- How to handle the enourmous amount of statistic which might get
collected.
- What trade-offs are feasable to consider to trim down the statistics
for every sample period.
- What trade-offs are feasable to consider when you consolidate, say, 6
x 5min samples into 1 x 30 min sample.
- What trade-offs are feasable to consider when graphing the data for a
specific timespan.
- What to call the tool?

Already now, there are commercial tools trying to do this out there
(I've come accross one of them).  They've faced some of the problems I
mention above in the "to discuss" list.

Your thoughts on this are welcome.

Best regards

/IlvJa

--
                (Jakob Ilves) <jakob.ilves at oracle.com>
             {Oracle Global IT, Network Management Group}
[Office as well as mobile phone: +46/8/477 3666 | Fax: +46/8/477 3572]
         - Intranet Home Page: http://jilves.se.oracle.com -



-- Attached file removed by Listar and put at URL below --
-- Type: text/x-vcard
-- Desc: Card for Jakob Ilves
-- Size: 444 bytes
-- URL : http://www.ee.ethz.ch/~slist/pantomime/18-jakob.ilves.vcf


--
Unsubscribe mailto:rrd-users-request at list.ee.ethz.ch?subject=unsubscribe
Help        mailto:rrd-users-request at list.ee.ethz.ch?subject=help
Archive     http://www.ee.ethz.ch/~slist/rrd-users
WebAdmin    http://www.ee.ethz.ch/~slist/lsg2.cgi