[rrd-users] Re: Strategies for data storage and graphing

[Ruttenberg, Tanya]

This is an interesting dilemma.

I've had a lot of thoughts about this since I started replying to your mail.

One of the main features of rrdtool is its ability to consolidate the data
over a period of time.  Or age out the data if you decide not to use RRA's.
You have to decide ahead of time the maximum amount of data you are ever
going to want to store there.

It seems from your email that you are planning to WORK AROUND that feature
and use rrd files as an alternative to your mysql database.  For instance,
you want to be able to see a graph of 90 days of data that reflects the data
in your mysql database. Is this correct?

If this is true, and you decide to use pre-created rrd files, it seems to me
you will have to create rrd files large enough to store the maximum amount
of data anyone could ever want to see a graph of.  If this is a possibility
in your environment, then I think suggestion 2 is the way to go. Or even
better, if you can save the pmacct data in the rrd files as you collect
it--at the same time you store it in your rdbms.  I don't know how you get
the data into mysql (not familiar with pmacct), but if it is put there via a
script you wrote, you can also write into it an rrdtool "adapter" that puts
the same data into rrd files at the time it is collected. 

If you cannot predict how much data someone is going to need (for instance
you never know if someone is going to want to view a graph with 2 years'
worth of data) then you have no idea how large to create your rrdfiles and
you have to somehow go with #1.  Though I don't see how to get around the
performance issue there.

There is not some other tool that extracts data from a database and
generates a graph directly from this data?  Though I agree rrdtool has
excellent graphing ability so I can see why you would want to leverage that.

Anyway, I'd be interested to hear how you resolve this.

Tanya Ruttenberg - RSIS Contractor
OTSO/DNE/NMPEB
tanya.ruttenberg at ssa.gov
410-965-9605


-----Original Message-----
From: rrd-users-bounce at list.ee.ethz.ch
[mailto:rrd-users-bounce at list.ee.ethz.ch] On Behalf Of Kenneth Kalmer
Sent: Tuesday, January 10, 2006 6:15 AM
To: rrd-users at list.ee.ethz.ch
Subject: [rrd-users] Strategies for data storage and graphing

Guys

This is my first post to the list, please excuse if me if it's not up to
scratch...

I currently collect network accounting data using pmacct and save it to
MySQL. Due to the nature of the network we need to keep a massive amount of
data for various reporting. I've been playing with rrdtool and a sample of
the data with great results.

I'd like to know what would be the best way to populate the rrd's from the
data in the database to create graphs. I know there are plenty of ways, but
speed is number one here. I'm also positive that I'm not first one facing
this problem...

Currently I'm faced with two possibilities, and would like any suggestions
on better ways of doing this.

1) Create the rrd, populate and graph in one smooth step. Works fine, but to
populate 90 days worth of 300 second step data can take up to 5 minutes...
Not desirable...

2) At set intervals, simply populate pre-created rrd's with the new data
since the last update, and when needed to graph add the remaining data
on-demand... Keeping track of the updates won't be difficult...
Haven't built this yet, but in theory it should yield much better
performance.

It boils down to needing rrd's excellent graphing capabilities, as well as
blistering speed when generating reports. This must be the same problem the
developers of cacti faced with all the data in MySQL, and rrdtools
graphing... Speaking of which, I'm about to tear up the cacti code looking
for insight, but any advise would be appreciated...

Best

--

Kenneth Kalmer
kenneth.kalmer at gmail.com

Folding at home stats
http://fah-web.stanford.edu/cgi-bin/main.py?qtype=userpage&username=kenneth%
2Ekalmer

--
Unsubscribe mailto:rrd-users-request at list.ee.ethz.ch?subject=unsubscribe
Help        mailto:rrd-users-request at list.ee.ethz.ch?subject=help
Archive     http://lists.ee.ethz.ch/rrd-users
WebAdmin    http://lists.ee.ethz.ch/lsg2.cgi

--
Unsubscribe mailto:rrd-users-request at list.ee.ethz.ch?subject=unsubscribe
Help        mailto:rrd-users-request at list.ee.ethz.ch?subject=help
Archive     http://lists.ee.ethz.ch/rrd-users
WebAdmin    http://lists.ee.ethz.ch/lsg2.cgi