[rrd-users] Re: VISIONARY: New tool to consider to create: a "DynamicData Set" tool...

[Jakob Ilves]

Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit

Hej

Tobias Oetiker wrote:

> Yesterday you sent me mail regarding [rrd-users] VISIONARY: New tool to...:
>
> JI> So what am I suggesting?  It's very short to express but probably not
> JI> trivial to achieve: write a tool which efficiently and accurately enough
> JI> might store, process and graph this kind of dynamicly changing data sets
> JI> and which supports the data consolidation over time MRTG/RRDtool uses
> JI> (store 5 min samples for 48 hours, store 30 min samples for 2 weeks
> JI> etc)..
>
> doing this with rrdtool should not be too difficult ... think about
> a directory tree with 3 levels ... each dir represents a byte of
> the contributing hosts IP address the last byte woud be the file
> name ... you can create rrd files as you go along ...

Well, if I do this on an intranet WAN link going from the Americas to EMEA
(Europe, Middle East, Africa) and we consider a large enterprise intranet, I
will have a HUGE number or rrd files being created as I have a large number of
destination hosts perhaps over 10 000.

But if I try to do some trade-off even before I start writing to the RRD:s...
OK, for each 5 minute period, process all the destination-datacount pairs so
that hosts which contribute to less than, say, 1% of the total amount of traffic
during those 5 minutes are put into the "low contributor RRD".  Then I'll only
have a maximum of 100 rrd files to access/create.  Problem might be though, that
the next 5 minutes, another large set of hosts might go above the "low
contributor" threshold and thus warrant to have RRD-files of their own.  Another
tradeoff could be to forget about 5 minute sample periods and instead use 30
minute studies at the start.

>
> my question would rather be, what reports do you want to generate
> from this data ... this will be the real challenge ... do you want

Well, if we extend the scope from not just destinations for traffic, I want the
tool to produce graphs with the same information as those produced by the
Netmetrix product, but better (of course ;-).  Netmetrix provides you with
statistics for a link such as graphs showing the distribution of:

   * protocol usage during the day
   * top talkers during the day
   * top listeners during the day
   * top conversation during the day.

(It also provides utilization graphs but that's not very intresting as RRDtool
VERY easily store and graphs such statistics, as we all know).

On each graph, the legend contains colors/patterns to distinguish the topmost
candidates for each category.  You can configure the reports to provide you with
top 10 talkers or top 5 talkers or top 40 talkers etc... any number you like
(too many messes up the graph, too few makes you miss some of the host).  As you
can tell, the legend and the color scheme cannot be decided on advance, as you
don't know which hosts/protocols/conversations will be the top 10 (or 5 or 40)
contributers to the traffic.  It has to be decide upon rendering the image.


>
> to store the whole matrix ... source AND destination ? this would
> make it probably unpractical with rrdtool, you might want to use a

But still RRDtool has some intresting features I really do like and which I
would like to see in the new tool and I think borrowing some design and code (if
you don't mind of course) could be a good idea.

>
> SQL database (postgresql) for this ... but again, the question what

Well, you could use a SQL database (like Oracle for instance) to actually store
the data, but still you could use a tool similar to rrdtool to create the data
collections, store and retrieve the data.  Actually, you could have something
very similar to an RRD but instead of storing values of how much utilization was
inbound at each sample period, you store a reference to a data structure which
contains what actually was seen during that sample period.

(When we talk about SQL databases, you could use such a one for RRDtool as well
if you want to).

>
> are you going to do with the data ?
>

Ok, here's an example.Well, consider the case you have a WAN link connecting an
subsidiary to the rest or the corporate intranet.  Suddenly some PC user decides
to do some gigantic download.  You want the graphs to display not only that the
link is near 100% utilized inbound, you also want to see in some graphs WHO is
using the major part of the link.

Another example is if you have a number of "favourite download servers"
somewhere on the intranet, and they become the source of the major portion of
traffic traversing the link, you want to the stats to indicate that (so you can
consider moving these servers or perhaps start replicate data to local download
servers.)

The two cases above does imply that not ALL the statistics about traffic over a
link is used, we are mainly intrested in what are the biggest contributors.
Which might imply that we may do trade-offs in statistic gathering and
processing.

> cheers
> tobi
>
> --
>  ______    __   _
> /_  __/_  / /  (_) Oetiker, Timelord & SysMgr @ EE-Dept ETH-Zurich
>  / // _ \/ _ \/ / TEL: +41(0)1-6325286  FAX:...1517  ICQ: 10419518
> /_/ \.__/_.__/_/ oetiker at ee.ethz.ch http://ee-staff.ethz.ch/~oetiker

--
                (Jakob Ilves) <jakob.ilves at oracle.com>
             {Oracle Global IT, Network Management Group}
[Office as well as mobile phone: +46/8/477 3666 | Fax: +46/8/477 3572]
         - Intranet Home Page: http://jilves.se.oracle.com -




-- Attached file removed by Listar and put at URL below --
-- Type: text/x-vcard
-- Desc: Card for Jakob Ilves
-- Size: 444 bytes
-- URL : http://www.ee.ethz.ch/~slist/pantomime/20-jakob.ilves.vcf


--
Unsubscribe mailto:rrd-users-request at list.ee.ethz.ch?subject=unsubscribe
Help        mailto:rrd-users-request at list.ee.ethz.ch?subject=help
Archive     http://www.ee.ethz.ch/~slist/rrd-users
WebAdmin    http://www.ee.ethz.ch/~slist/lsg2.cgi