[rrd-users] Effects of RRAs on CPU, disk I/O, and memory resources

[Reinhard Scheck]

On 24.09.2010 20:44, Matt Kassawara wrote:
> My organization wants to record more detailed information for longer 
> periods of time without consuming excessive resources on the server 
> performing RRD operations and storing RRD files.  I know RRD files 
> maintain the same size on disk after creation.  However, I'm unsure how 
> the number of RRAs and RRA step/row values affect CPU, disk I/O, and 
> memory resources particularly during update operations and periodic 
> internal consolidations.  Further complicating the situation, each 
> graphing application manages RRD files differently.  For example, Zenoss 
> creates one RRD file with a single DS and associated RRAs for each OID 
> retrieved from a device.  Retrieving just a few OIDs from a large number 
> of devices results in Zenoss operating on many small RRD files per 
> update cycle.  I haven't found a way to configure it otherwise.  On the 
> other hand, Cacti creates one RRD file with a variable number of DS 
> based on templates.  I can configure and implement these templates to 
> perform roughly the opposite management style of Zenoss by increasing 
> the size and complexity of each RRD file, but reducing the overall 
> number of them.  I know discussion of Zenoss vs. Cacti wanders 
> off-topic, but in this case providing the methods they use for managing 
> RRD files may greatly affect answers to my primary question.
> 
> I currently use Zenoss to retrieve about 11000 OIDs every 5 minutes. 
> Each of the 11000 RRD files contains 4 RRAs providing data for about 2 
> days at 5 minute intervals (essentially no consolidation), 2 weeks at 30 
> minute intervals, 50 days at 2 hour intervals, and 600 days at 1 day 
> intervals.
While I can't provide a formulae or something exactly answering your question, I
have to state that 11000 files of either type do not pose a significant problem
to a mid sized server.
This is derived from my personal experience with Cacti and, as a forum
moderator, deduced from user responses.
So, to be honest, we did not bother yet.

>From my understanding of rrdtool, I'm quite sure that updates are performed on a
single/some few blocks of the data structure only.
There has been a discussion on taking advantage of "fadvise" in Linux OS to
prevent unwanted read ahead caching that does not serve rrdtool updates well;
pointing again into the same direction.

And, with rrdtool 1.4.x, there's the rrdcached that will again boost file
updates (in the cacti ecosystem, you would choose "boost" which does almost the
same).
Reinhard