[rrd-developers] Re: rrd::graph memory usage patch

[Holger Jahn]

Ole,

What exactly do you mean by "slow"? 

Out of my experience, generating graphs out of RRD file with 11MB in size only should be done within less than 50ms.

You may want to have a look at the I/O utilisation of the disk your RRD files live on.

Holger

-----Original Message-----
From: obh at telenor.net [mailto:obh at telenor.net] 
Sent: Friday, 9 December 2005 10:06 p.m.
To: Holger Jahn
Cc: rrd-developers at list.ee.ethz.ch
Subject: Re: [rrd-developers] rrd::graph memory usage patch


This patch was interesting.  We are using 64 SNMP variables in each RRD
file to balance the load on IO subsystem versus the speed of drawing
the graph. Still, when files are 11MB it is somewhat slow to draw graph
- but tolerable.

Ole Bjørn Hessen
Telenor.



"Holger Jahn" <hjahn at woosh.com> writes: > Tobi & Gents, > Some days ago
I bumped into an issue with rrd::graph that encouraged me > to
reactivate my C skills..... ;-)
> 
> The issue:
> 
> We are using RRD files which are 1.8GB each (800 RRAs per file with 5
> mins resolution for one year), which get updated every minute. When I
> generate a graph out of the RRAs it works file with time periods "Now =>
> 2 Weeks back". Every attempt to generate graphs that span a larger time
> period, such as half a year, ends up pushing our server into memory
> swapping.
> 
> Splitting the RRD files into many smaller ones is not an option, since
> this would push the I/O subsystem on our server against the wall
> (currently we use a total of 36 GB data stored in RRD databases,
> reflecting 144,000 SNMP variables).
> 
> After having had a deep look into the sources I found that rrd_fetch()
> basically reads the entire file into memory (in case of a 1 year
> graphing period), even if only one RRA is required to be graphed.
> 
> The solution:
> 
> I modified some files in the source code of 1.2.10 to only read the CDPs
> that are required to be graphed, effectively skipping the RRAs we don't
> need.
> 
> The code is a crude hack, but working. Basically I have copied all lines
> I have changed and commented out the original sections (using C++ //
> comment markers to make it easy to identify the original parts I have
> changed).
> 
> Since I have modified the rrd_fetch(), data_fetch() and rrd_graph()
> functions, it might be
> highly likely that I have introduced many side-effects into other parts
> of RRDtool. Thus it would be great if someone of the RRD Gods would
> revise my patch.
> 
> Cheers,
> 
> Holger
> 
> 
> -- Attached file removed by Ecartis and put at URL below --
> -- Type: application/octet-stream
> -- Desc: rrdtool-1.2.11.patch
> -- Size: 18k (18522 bytes)
> -- URL : http://lists.ee.ethz.ch/p/02-rrdtool-1.2.11.patch
> 
> 
> --
> Unsubscribe mailto:rrd-developers-request at list.ee.ethz.ch?subject=unsubscribe
> Help        mailto:rrd-developers-request at list.ee.ethz.ch?subject=help
> Archive     http://lists.ee.ethz.ch/rrd-developers
> WebAdmin    http://lists.ee.ethz.ch/lsg2.cgi
> 



Ole Bjørn Hessen,
NMS-IP, PF-Nett, Telenor Networks


--
Unsubscribe mailto:rrd-developers-request at list.ee.ethz.ch?subject=unsubscribe
Help        mailto:rrd-developers-request at list.ee.ethz.ch?subject=help
Archive     http://lists.ee.ethz.ch/rrd-developers
WebAdmin    http://lists.ee.ethz.ch/lsg2.cgi