[rrd-users] Re: new log2rrd beta - handles MAX values, etc.

[Dave Plonka]

On Mon, Jun 26, 2000 at 12:22:49PM +0200, Rainer Bawidamann wrote:
> 
> In article <20000623165619.A10553 at doit.wisc.edu>,
> 	plonka at doit.wisc.edu (Dave Plonka) writes:
> > A coworker and I have modified "log2rrd" - the utility to convert from
> > MRTG log files to RRD files - to handle the MAX values as well.  I just
> > couldn't bring myself to discard years of MAX data from all my ".log"
> > files.
> 
> Great! Thanks for your work!
> 
> Some comments:
> 
> You create the rrd files with the rras (steps:rows, MAX/AVERAGE each)
>        1:600 / 6:600 / 24:600 / 288:600
> (altough the documentation says 732 rows for the last rra). But the mrtg
> log files contain older data: The rras of a rrd file overlap (all end
> "now"), the "rras" of a mrtg log file not (they "append"). I think The
> Right Sizes (TM) for a mrtg-compatible rrd file is:
> 
>        1:600 / 6:700 / 24:775 / 288:797

Hmm, I'll have to check into that then.  In its current form, I blindly
trusted what the previous log2rrd authors had done with respect to RRA
dimensions since the comments led me to believe that some thought had
been put into it.

I *think* that we want log2rrd to preserve the RRA sizes that store
exactly the data that the ".log" file contains, not (necessarily) what
it would create if one initially sets "UseRRDTool Yes".  Which sizes
are you giving me?

> The next obvious question is why you don't create a xml file from
> scratch instead using rrdtool to create a rrd file, than a xml file ...
> - but I know the answer (you would have to do the maximum/average
> calculation for some values yourself)  :-/

Yes ;^), refer to my answers to, "Why didn't you write this in assembler?",
"Why didn't you reinvent the wheel?", etc.

Actually, no, it's just that "rrdtool dump" writes nice XML for us, and
makes sure that "rrdtool restore" will be able to make sense of it as
well.  It would probably a number of times the time to hack out and
testing a version that builds the XML from scratch.  I'm using the
power of RRDTOOL and XML::Parser and such so I don't have to do the
dirty work.

If your thought is that XML::Parser is slow - that may be.  [As an
aside, that was mentioned in Tim Bray's report on "Perl Whirl":
http://www.oreillynet.com/pub/a/network/2000/06/06/geekcruise.html.]
But if log2rrd can convert my hundreds of ".log" files even in hours
its fine with me.  Anyway, remember, log2rrd is probably going to be
run once, potentially by many users, but probably only once each.

Also, my example of how to manipulate RRDs by using XML::Parser and
XML::Simple (albeit somewhat buried in log2rrd specific code) can
hopefully enable others to build other utilities to massage RRD files.

For instance, this method could be used to "tune" any part of the RRD
or add DSes or extend RRAs (most easily the last RRA - of course one
must be very careful to produce an output XML that makes sense before
restoring it with "rrdtool restore".)

> And what about reading a mrtg config file to look for
> the log files and the datasourcetype? ;-)

You mean look for the "gauge", "absolute", etc. options...  hmm...
I might be willing to add this - if it could be done reasonably and
quickly.

That could be done with a wrapper script though, external to log2rrd.
I'm not sure it's worth the effort of my independently parsing the MRTG
config file just to glean those details.  I have added the ability for
the caller to specify the DS type on the log2rrd command line, and I
would assert that it is well within the capability of anyone who has
delved into using those MRTG options.  Also, now log2rrd defaults to
COUNTER - just like MRTG does - so it should do the right thing for
most targets/users.  (Which, of course is almost never the Right Thing,
absolutely.)

Is there some other benefit I could get from parsing the MRTG config
file that would make it a bigger win?

Thanks for the feedback.

Dave

-- 
plonka at doit.wisc.edu  http://net.doit.wisc.edu/~plonka  ARS:N9HZF  Madison, WI