<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div><div>So we had about 38k rrd files being updated every 15 seconds. That's a heavy I/O load, and we need newer bigger hardware for this. When I wrote the script to go through and resize the RRD files we saw each one take between 2-6 seconds. (5 different resizes on each file) Since every original file and final file were the same size (same action on each) the difference in timing would be due to I/O load on the system. I tried to be smart and only write to files which had been written to less than 5 seconds before but that really didn't seem to work as well as I hoped. The good news is that either RRD has some integral locking, or the RRD resize just didn't matter for update because we saw zero corruption after I disabled mmap.</div><div><br></div><div>CentOS 5.6 with 2TB, quad core, 24GB memory and SATA RAID-1. Yes, under-spec for this job :)</div><div><br></div><div>On Jul 17, 2012, at 2:22 PM, Ryan Kubica wrote:</div><blockquote type="cite"><div><div style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); font-family: arial, helvetica, sans-serif; font-size: 10pt; position: static; z-index: auto; "><div style="font-family: arial, helvetica, sans-serif; font-size: 10pt; "><span>To your 'this system is too critical to pause' ... rrdtool is very fast ... and this is my apps writer:</span></div><div style="font-family: arial, helvetica, sans-serif; font-size:
10pt; "><span><br></span></div><div><span><div><font size="2">Tue Jul 17 20:58:16:525 2012 - pid:614 - <span style="font-weight: bold;">1031706</span>/0/0/0 updates/errors/saved/creates completed in 6.514 seconds at a rate of <span style="font-weight: bold;">158376.232</span> a second.</font></div><div style="font-family: arial, helvetica, sans-serif; font-size: 10pt; "><br></div><div style="font-family: arial, helvetica, sans-serif; font-size: 10pt; ">In that write-cycle, 1 million updates (<span style="font-weight: bold;">no</span> rrdcached) at 158k updates per second; which is a bit slower than prod runs since this output is from a load test rrd server.</div><div style="font-family: arial, helvetica, sans-serif; font-size: 10pt; "><br></div><div style="font-family: arial, helvetica, sans-serif; font-size: 10pt; ">a resize takes milliseconds (16 milliseconds for my rrd test I gave you); so an inline task in a writer can do these with no problem;
unless you want to resize many thousands all at once which may(probably) incur IO penalty/latency ... to that, you just reduce the rate of resizes per minute (which my app does too.)</div><div style="font-family: arial, helvetica, sans-serif; font-size: 10pt; "><br></div><div style="font-family: arial, helvetica, sans-serif; font-size: 10pt; ">All apps are different; it's just food for thought but likely easiest/safest to resize when it's not being written to, and once the resize is done a rename is atomic.</div><div style="font-family: arial, helvetica, sans-serif; font-size: 10pt; "><br></div><div style="font-family: arial, helvetica, sans-serif; font-size: 10pt; ">I'm running OEL6 (RHEL6.1) 2.6.32 on an Intel based 12-core X5650 @ 2.67GHz, 72GB memory, 16 1TB 7400RPM SATA RAID-10; it's a nice server ... stores 1.5 million rrds w/step 60 every minute.</div><div style="font-family: arial, helvetica, sans-serif; font-size: 10pt;
"><br></div><div><div><span style="font-size: 13px;">[~]$ /apps/epic/rrdtool/bin/rrdtool -v</span></div><div><span style="font-size: 13px;">RRDtool 1.4.7 Copyright 1997-2012 by Tobias Oetiker <<a href="mailto:tobi@oetiker.ch">tobi@oetiker.ch</a>></span></div><div><span style="font-size: 13px;"> Compiled Jan 24 2012 21:56:18</span></div><div style="font-family: arial, helvetica, sans-serif; font-size: 10pt; "><br></div><div style="font-family: arial, helvetica, sans-serif; font-size: 10pt; "><br></div></div><div style="font-family: arial, helvetica, sans-serif; font-size: 10pt; ">-Ryan</div><div style="font-family: arial, helvetica, sans-serif; font-size: 10pt; "><br></div></span></div><div style="font-family: arial, helvetica, sans-serif; font-size: 10pt; "><br></div> <div style="font-family: arial, helvetica, sans-serif; font-size: 10pt; "> <div style="font-family: 'times new roman', 'new york', times, serif; font-size:
12pt; "> <div dir="ltr"> <font size="2" face="Arial"> <hr size="1"> <b><span style="font-weight:bold;">From:</span></b> Jo Rhett <<a href="mailto:jrhett@netconsonance.com">jrhett@netconsonance.com</a>><br> <b><span style="font-weight: bold;">To:</span></b> Ryan Kubica <<a href="mailto:kubicaryan@yahoo.com">kubicaryan@yahoo.com</a>> <br><b><span style="font-weight: bold;">Cc:</span></b> Steve Shipway <<a href="mailto:s.shipway@auckland.ac.nz">s.shipway@auckland.ac.nz</a>>; "<a href="mailto:rrd-users@lists.oetiker.ch">rrd-users@lists.oetiker.ch</a>" <<a href="mailto:rrd-users@lists.oetiker.ch">rrd-users@lists.oetiker.ch</a>> <br> <b><span style="font-weight: bold;">Sent:</span></b> Tuesday, July 17, 2012 10:27 AM<br> <b><span style="font-weight: bold;">Subject:</span></b> Re: [rrd-users] rrdtool resize corrupts timestamps in both 1.3.9 and 1.4.7<br> </font> </div> <br>
<div id="yiv1895015146"><div><div><div>On Jul 16, 2012, at 10:28 PM, Ryan Kubica wrote:</div><blockquote type="cite"><div><div style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); font-family: arial, helvetica, sans-serif; font-size: 10pt; "><div style="font-family: arial, helvetica, sans-serif; font-size: 10pt; ">I've never run
into this issue before (and store a <span style="font-weight:bold;">lot</span> of data) and on 64bit hosts, but:</div><div style="font-family: arial, helvetica, sans-serif; font-size: 10pt; "><br></div><div style="font-family: arial, helvetica, sans-serif; font-size: 10pt; "><span class="yiv1895015146Apple-tab-span" style="white-space:pre;"> </span>a) I always compile rrdtool not use stock OS rpm</div></div></div></blockquote><div><br></div>This isn't a common situation -- most people use RPMs, and that means that this is broken for most people. Also we are using a package that bundles a binary image of rrdtool, another situation where many people won't or can't change it out.</div><div><br><blockquote type="cite"><div><div style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); font-family: arial, helvetica, sans-serif; font-size: 10pt; "><div style="font-family: arial, helvetica, sans-serif; font-size: 10pt; "><span class="yiv1895015146Apple-tab-span" style="white-space:pre;"> </span>b) It is compiled with mmap()</div><div style="font-family: arial, helvetica, sans-serif; font-size: 10pt; "><span class="yiv1895015146Apple-tab-span" style="white-space:pre;"> </span>c) I only resize an rrdtool file when it's not being written to (my apps write daemon does the resize in place as a task.)</div></div></div></blockquote><div><br></div>That's not a choice I get. This system is too critical to stop updates long enough for that to happen. (although I have been digging around in the code to see if there is any locking mechanism that rrdtool will honor)</div><div><br><blockquote type="cite"><div><div style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); font-family: arial, helvetica, sans-serif; font-size: 10pt; "><div style="font-family: arial, helvetica, sans-serif; font-size: 10pt; ">and it works for resizes just fine (just tested manually), I grew RRA 0 by
2x.</div><div style="font-family: arial, helvetica, sans-serif; font-size: 10pt; "><br></div><div><div><span style="font-size:13px;">[tmp]$ grep 1340865780 *xml</span></div><div><span style="font-size:13px;">Epic_io_90.xml:<span class="yiv1895015146Apple-tab-span" style="white-space:pre;"> </span><!-- 2012-06-28 06:43:00 UTC / 1340865780 --> <row><v>9.7033939036e+04</v></row></span></div><div><span style="font-size:13px;">resize.xml:<span class="yiv1895015146Apple-tab-span" style="white-space:pre;"> </span><!-- 2012-06-28 06:43:00 UTC / 1340865780 --> <row><v>9.7033939036e+04</v></row></span></div><div><span style="font-size:13px;"><br></span></div><div><span style="font-size:13px;">[tmp]$ grep 1340865840 *xml</span></div><div><span style="font-size:13px;">Epic_io_90.xml:<span class="yiv1895015146Apple-tab-span" style="white-space:pre;"> </span><!-- 2012-06-28 06:44:00 UTC / 1340865840
--> <row><v>9.7633476991e+04</v></row></span></div><div><span style="font-size:13px;">resize.xml:<span class="yiv1895015146Apple-tab-span" style="white-space:pre;"> </span><!-- 2012-06-28 06:44:00 UTC / 1340865840 --> <row><v>9.7633476991e+04</v></row></span></div></div></div></div></blockquote><div><br></div>Curious, what OS / kernel are you using?</div><div><br></div><div>
<span class="yiv1895015146Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; font-size: medium; "><span class="yiv1895015146Apple-style-span" style="font-size:12px;"><div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;"><font size="3" style="font-style: normal; font-variant: normal; font-weight: normal; font-size: 12px; line-height: normal; font-family: Helvetica; ">-- </font></div><div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;"><font size="3" style="font-style: normal; font-variant: normal; font-weight: normal; font-size: 12px; line-height: normal; font-family: Helvetica; ">Jo Rhett</font></div></span><span class="yiv1895015146Apple-style-span" style="font-size:12px;">Net Consonance : </span><span class="yiv1895015146Apple-style-span" style="font-size:12px;">net philanthropy to improve open source and internet projects.</span><br><span class="yiv1895015146Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; "><div style="word-wrap:break-word;"><div><div><span class="yiv1895015146Apple-style-span" style="font-size:12px;"><div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;"><br></div></span></div></div></div></span></span><br class="yiv1895015146Apple-interchange-newline">
</div>
<br></div></div><br><br> </div> </div> </div></div></blockquote></div><br><div>
<span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><span class="Apple-style-span" style="font-size: 12px; "><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Helvetica" size="3" style="font: normal normal normal 12px/normal Helvetica; ">-- </font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Helvetica" size="3" style="font: normal normal normal 12px/normal Helvetica; ">Jo Rhett</font></div></span><span class="Apple-style-span" style="font-size: 12px; ">Net Consonance : </span><span class="Apple-style-span" style="font-size: 12px; ">net philanthropy to improve open source and internet projects.</span><br><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div><div><span class="Apple-style-span" style="font-size: 12px; "><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><br></div></span></div></div></div></span></span><br class="Apple-interchange-newline">
</div>
<br></body></html>