[rrd-users] rrd resize speed

[Jo Rhett]

So we had about 38k rrd files being updated every 15 seconds. That's a heavy I/O load, and we need newer bigger hardware for this. When I wrote the script to go through and resize the RRD files we saw each one take between 2-6 seconds. (5 different resizes on each file) Since every original file and final file were the same size (same action on each) the difference in timing would be due to I/O load on the system.  I tried to be smart and only write to files which had been written to less than 5 seconds before but that really didn't seem to work as well as I hoped. The good news is that either RRD has some integral locking, or the RRD resize just didn't matter for update because we saw zero corruption after I disabled mmap.

CentOS 5.6 with 2TB, quad core, 24GB memory and SATA RAID-1.  Yes, under-spec for this job :)

On Jul 17, 2012, at 2:22 PM, Ryan Kubica wrote:
> To your 'this system is too critical to pause' ... rrdtool is very fast ... and this is my apps writer:
> 
> Tue Jul 17 20:58:16:525 2012 - pid:614 - 1031706/0/0/0 updates/errors/saved/creates completed in 6.514 seconds at a rate of 158376.232 a second.
> 
> In that write-cycle, 1 million updates (no rrdcached) at 158k updates per second; which is a bit slower than prod runs since this output is from a load test rrd server.
> 
> a resize takes milliseconds (16 milliseconds for my rrd test I gave you); so an inline task in a writer can do these with no problem; unless you want to resize many thousands all at once which may(probably) incur IO penalty/latency ... to that, you just reduce the rate of resizes per minute (which my app does too.)
> 
> All apps are different; it's just food for thought but likely easiest/safest to resize when it's not being written to, and once the resize is done a rename is atomic.
> 
> I'm running OEL6 (RHEL6.1) 2.6.32 on an Intel based 12-core X5650 @ 2.67GHz, 72GB memory, 16 1TB 7400RPM SATA RAID-10; it's a nice server ... stores 1.5 million rrds w/step 60 every minute.
> 
> [~]$ /apps/epic/rrdtool/bin/rrdtool -v
> RRDtool 1.4.7  Copyright 1997-2012 by Tobias Oetiker <tobi at oetiker.ch>
>                Compiled Jan 24 2012 21:56:18
> 
> 
> -Ryan
> 
> 
> From: Jo Rhett <jrhett at netconsonance.com>
> To: Ryan Kubica <kubicaryan at yahoo.com> 
> Cc: Steve Shipway <s.shipway at auckland.ac.nz>; "rrd-users at lists.oetiker.ch" <rrd-users at lists.oetiker.ch> 
> Sent: Tuesday, July 17, 2012 10:27 AM
> Subject: Re: [rrd-users] rrdtool resize corrupts timestamps in both 1.3.9 and 1.4.7
> 
> On Jul 16, 2012, at 10:28 PM, Ryan Kubica wrote:
>> I've never run into this issue before (and store a lot of data) and on 64bit hosts, but:
>> 
>> 	a) I always compile rrdtool not use stock OS rpm
> 
> This isn't a common situation -- most people use RPMs, and that means that this is broken for most people.  Also we are using a package that bundles a binary image of rrdtool, another situation where many people won't or can't change it out.
> 
>> 	b) It is compiled with mmap()
>> 	c) I only resize an rrdtool file when it's not being written to (my apps write daemon does the resize in place as a task.)
> 
> That's not a choice I get. This system is too critical to stop updates long enough for that to happen. (although I have been digging around in the code to see if there is any locking mechanism that rrdtool will honor)
> 
>> and it works for resizes just fine (just tested manually), I grew RRA 0 by 2x.
>> 
>> [tmp]$ grep 1340865780 *xml
>> Epic_io_90.xml:			<!-- 2012-06-28 06:43:00 UTC / 1340865780 --> <row><v>9.7033939036e+04</v></row>
>> resize.xml:			<!-- 2012-06-28 06:43:00 UTC / 1340865780 --> <row><v>9.7033939036e+04</v></row>
>> 
>> [tmp]$ grep 1340865840 *xml
>> Epic_io_90.xml:			<!-- 2012-06-28 06:44:00 UTC / 1340865840 --> <row><v>9.7633476991e+04</v></row>
>> resize.xml:			<!-- 2012-06-28 06:44:00 UTC / 1340865840 --> <row><v>9.7633476991e+04</v></row>
> 
> Curious, what OS / kernel are you using?
> 
> -- 
> Jo Rhett
> Net Consonance : net philanthropy to improve open source and internet projects.
> 
> 
> 
> 
> 

-- 
Jo Rhett
Net Consonance : net philanthropy to improve open source and internet projects.



-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.oetiker.ch/pipermail/rrd-users/attachments/20120717/ae567fba/attachment.htm