[rrd-users] Re: Strategies for data storage and graphing

[Kenneth Kalmer]

On 1/10/06, Ruttenberg, Tanya <Tanya.Ruttenberg at ssa.gov> wrote:
> This is an interesting dilemma.

Have to agree with you here...

> I've had a lot of thoughts about this since I started replying to your mail.
>
> One of the main features of rrdtool is its ability to consolidate the data
> over a period of time.  Or age out the data if you decide not to use RRA's.
> You have to decide ahead of time the maximum amount of data you are ever
> going to want to store there.

This is where option #1 comes in handy, you can generate an rrd of the
required size on the fly.

> It seems from your email that you are planning to WORK AROUND that feature
> and use rrd files as an alternative to your mysql database.  For instance,
> you want to be able to see a graph of 90 days of data that reflects the data
> in your mysql database. Is this correct?

Never thought about it being a work around, but it appears so. Yip,
all the data resides in a MySQL database.

> If this is true, and you decide to use pre-created rrd files, it seems to me
> you will have to create rrd files large enough to store the maximum amount
> of data anyone could ever want to see a graph of.  If this is a possibility
> in your environment, then I think suggestion 2 is the way to go. Or even
> better, if you can save the pmacct data in the rrd files as you collect
> it--at the same time you store it in your rdbms.  I don't know how you get
> the data into mysql (not familiar with pmacct), but if it is put there via a
> script you wrote, you can also write into it an rrdtool "adapter" that puts
> the same data into rrd files at the time it is collected.

pmacct does the dirty work for me here, it saves the accounting
directly to MySQL. I can however configure pmacct to execute a script
upon each write to the database, so I can update the rrd's like that.
pmacct is a great tool for collecting data for network accounting
purposes, check out http://www.ba.cnr.it/~paolo/pmacct/. Using some
smart tricks you can populate rrd's directly using it's memory plugin
(instead of MySQL), but I require the MySQL route here...

> If you cannot predict how much data someone is going to need (for instance
> you never know if someone is going to want to view a graph with 2 years'
> worth of data) then you have no idea how large to create your rrdfiles and
> you have to somehow go with #1.  Though I don't see how to get around the
> performance issue there.

Agreed...

> There is not some other tool that extracts data from a database and
> generates a graph directly from this data?  Though I agree rrdtool has
> excellent graphing ability so I can see why you would want to leverage that.

The reporting system is coded in PHP, and there are a good couple of
PHP classes that generate all kinds of statistical graphs, charts,
etc.. The one I've found the best so far is a library called jpGraph.
But it chokes on creating a graph with so much data and the memory
consumption in PHP is tremendous when doing this. This triggered my
investigations into an alternative, hence rrd.

> Anyway, I'd be interested to hear how you resolve this.

I'm digging around the cacti code to see how they handle it, but it
seems that option #2 is definitely the way to go here...

> Tanya Ruttenberg - RSIS Contractor
> OTSO/DNE/NMPEB
> tanya.ruttenberg at ssa.gov
> 410-965-9605

Kenneth

> -----Original Message-----
> From: rrd-users-bounce at list.ee.ethz.ch
> [mailto:rrd-users-bounce at list.ee.ethz.ch] On Behalf Of Kenneth Kalmer
> Sent: Tuesday, January 10, 2006 6:15 AM
> To: rrd-users at list.ee.ethz.ch
> Subject: [rrd-users] Strategies for data storage and graphing
>
> Guys
>
> This is my first post to the list, please excuse if me if it's not up to
> scratch...
>
> I currently collect network accounting data using pmacct and save it to
> MySQL. Due to the nature of the network we need to keep a massive amount of
> data for various reporting. I've been playing with rrdtool and a sample of
> the data with great results.
>
> I'd like to know what would be the best way to populate the rrd's from the
> data in the database to create graphs. I know there are plenty of ways, but
> speed is number one here. I'm also positive that I'm not first one facing
> this problem...
>
> Currently I'm faced with two possibilities, and would like any suggestions
> on better ways of doing this.
>
> 1) Create the rrd, populate and graph in one smooth step. Works fine, but to
> populate 90 days worth of 300 second step data can take up to 5 minutes...
> Not desirable...
>
> 2) At set intervals, simply populate pre-created rrd's with the new data
> since the last update, and when needed to graph add the remaining data
> on-demand... Keeping track of the updates won't be difficult...
> Haven't built this yet, but in theory it should yield much better
> performance.
>
> It boils down to needing rrd's excellent graphing capabilities, as well as
> blistering speed when generating reports. This must be the same problem the
> developers of cacti faced with all the data in MySQL, and rrdtools
> graphing... Speaking of which, I'm about to tear up the cacti code looking
> for insight, but any advise would be appreciated...
>
> Best
>
> --
>
> Kenneth Kalmer
> kenneth.kalmer at gmail.com
>
> Folding at home stats
> http://fah-web.stanford.edu/cgi-bin/main.py?qtype=userpage&username=kenneth%
> 2Ekalmer
>
> --
> Unsubscribe mailto:rrd-users-request at list.ee.ethz.ch?subject=unsubscribe
> Help        mailto:rrd-users-request at list.ee.ethz.ch?subject=help
> Archive     http://lists.ee.ethz.ch/rrd-users
> WebAdmin    http://lists.ee.ethz.ch/lsg2.cgi
>


--

Kenneth Kalmer
kenneth.kalmer at gmail.com

Folding at home stats
http://fah-web.stanford.edu/cgi-bin/main.py?qtype=userpage&username=kenneth%2Ekalmer

--
Unsubscribe mailto:rrd-users-request at list.ee.ethz.ch?subject=unsubscribe
Help        mailto:rrd-users-request at list.ee.ethz.ch?subject=help
Archive     http://lists.ee.ethz.ch/rrd-users
WebAdmin    http://lists.ee.ethz.ch/lsg2.cgi