[rrd-users] rrdtool resize corrupts timestamps in both 1.3.9 and 1.4.7

[Ryan Kubica]

My 'but' list was just as a point of reference that my situation is different than yours ... my concern came from you finding something that seemed like corruption, so I wanted to know for myself what it was too and thus check myself if it was reproducible.  Your right, not everyone recompiles.

To your 'this system is too critical to pause' ... rrdtool is very fast ... and this is my apps writer:

Tue Jul 17 20:58:16:525 2012 - pid:614 - 1031706/0/0/0 updates/errors/saved/creates completed in 6.514 seconds at a rate of 158376.232 a second.

In that write-cycle, 1 million updates (no rrdcached) at 158k updates per second; which is a bit slower than prod runs since this output is from a load test rrd server.

a resize takes milliseconds (16 milliseconds for my rrd test I gave you); so an inline task in a writer can do these with no problem; unless you want to resize many thousands all at once which may(probably) incur IO penalty/latency ... to that, you just reduce the rate of resizes per minute (which my app does too.)

All apps are different; it's just food for thought but likely easiest/safest to resize when it's not being written to, and once the resize is done a rename is atomic.

I'm running OEL6 (RHEL6.1) 2.6.32 on an Intel based 12-core X5650 @ 2.67GHz, 72GB memory, 16 1TB 7400RPM SATA RAID-10; it's a nice server ... stores 1.5 million rrds w/step 60 every minute.

[~]$ /apps/epic/rrdtool/bin/rrdtool -v
RRDtool 1.4.7  Copyright 1997-2012 by Tobias Oetiker <tobi at oetiker.ch>
               Compiled Jan 24 2012 21:56:18


-Ryan



________________________________
 From: Jo Rhett <jrhett at netconsonance.com>
To: Ryan Kubica <kubicaryan at yahoo.com> 
Cc: Steve Shipway <s.shipway at auckland.ac.nz>; "rrd-users at lists.oetiker.ch" <rrd-users at lists.oetiker.ch> 
Sent: Tuesday, July 17, 2012 10:27 AM
Subject: Re: [rrd-users] rrdtool resize corrupts timestamps in both 1.3.9 and 1.4.7
 

On Jul 16, 2012, at 10:28 PM, Ryan Kubica wrote:
I've never run into this issue before (and store a lot of data) and on 64bit hosts, but:
>
>
>a) I always compile rrdtool not use stock OS rpm
This isn't a common situation -- most people use RPMs, and that means that this is broken for most people.  Also we are using a package that bundles a binary image of rrdtool, another situation where many people won't or can't change it out.


b) It is compiled with mmap()
>c) I only resize an rrdtool file when it's not being written to (my apps write daemon does the resize in place as a task.)
That's not a choice I get. This system is too critical to stop updates long enough for that to happen. (although I have been digging around in the code to see if there is any locking mechanism that rrdtool will honor)


and it works for resizes just fine (just tested manually), I grew RRA 0 by 2x.
>
>
>[tmp]$ grep 1340865780 *xml
>Epic_io_90.xml:<!-- 2012-06-28 06:43:00 UTC / 1340865780 --> <row><v>9.7033939036e+04</v></row>
>resize.xml:<!-- 2012-06-28 06:43:00 UTC / 1340865780 --> <row><v>9.7033939036e+04</v></row>
>
>
>[tmp]$ grep 1340865840 *xml
>Epic_io_90.xml:<!-- 2012-06-28 06:44:00 UTC / 1340865840 --> <row><v>9.7633476991e+04</v></row>
>resize.xml:<!-- 2012-06-28 06:44:00 UTC / 1340865840 --> <row><v>9.7633476991e+04</v></row>
Curious, what OS / kernel are you using?

-- 
Jo RhettNet Consonance : net philanthropy to improve open source and internet projects.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.oetiker.ch/pipermail/rrd-users/attachments/20120717/1071da9e/attachment-0001.htm