<div class="gmail_quote">On Thu, Oct 21, 2010 at 8:50 PM, Steve Shipway <span dir="ltr"><<a href="mailto:s.shipway@auckland.ac.nz">s.shipway@auckland.ac.nz</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div lang="EN-NZ" link="blue" vlink="purple">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">The corrupted file ends up the correct size; however the entire
file is filled with zeroes (fortunately, we archive our RRD files nightly so I
can go back and retrieve the last uncorrupted version plus the corrupted
version)</span></p></div></div></blockquote><div><br></div><div>Strange... The failure mode for rrd_open() unmaps and closes the file... that's about it. I'm not sure how it could zero the file like that.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div lang="EN-NZ" link="blue" vlink="purple"><div><p class="MsoNormal"><span class="Apple-style-span" style="font-size: 15px; color: rgb(31, 73, 125); "> </span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">The system is not (normally) memory or process-constrained;
there is in fact nothing to speak of running apart from apache and the
rrdcached daemon. The rrdinfo response is ‘not an RRD file’,
since it doesn’t have the RRD header.</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"> </span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">It has run fine for a whole week at these rates before the
problem hit; so that’s why I think it might be a leak in the RRD
functions (which would of course not show up in a non-daemon situation).
We use the remote update, info and (occasionally) create via the TCP socket;
plus the info, last, flush and fetch via the UNIX socket.</span><span class="Apple-style-span" style="font-size: 15px; color: rgb(31, 73, 125); "> </span></p></div></div></blockquote><div><br></div><div>My workload is all UPDATE and FLUSH and I'm not seeing any problems. It's possible that the newer code (info, create) has a leak that I haven't caught yet in production.</div>
<div><br></div><div>Could you show me:</div><div><br></div><div> - the output of 'stats' from your daemon</div><div> - "rrdtool info" from an RRD that's typical of your workload</div><div> - the args you're using when starting the rrdcached daemon </div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div lang="EN-NZ" link="blue" vlink="purple"><div>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">The build is the absolute latest r2136 .</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"> </span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">The memory usage of the rrdcached process is definitely
increasing; however that may also be due to the number of items in the
queue? It is currently at 768m virtual, 560m physical (17% usage) which
seems somewhat high to me, even for 20,000+ RRD files. Eventually it will
hit address-space limits (this is a 32bit RHEL5 box with 4G physical memory)</span></p></div></div></blockquote><div><br></div><div>My rrdcached runs around 2GB. That's with about 350k RRDs and 72 cached values per RRD. So, your memory utilization does look high.</div>
<div><span class="Apple-style-span" style="font-size: 15px; color: rgb(31, 73, 125); "> </span></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div lang="EN-NZ" link="blue" vlink="purple">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Unfortunately I don’t have any of the nice developer tools
for tracking memory leaks…</span></p></div></div></blockquote><div><br></div><div>You could install "valgrind" and run the daemon under that for a while. The daemon should be compiled with debugging symbols (-g) and not stripped in this case. i.e.</div>
<div><br></div><div>% valgrind --leak-check=full --show-reachable=yes rrdcached -args blah blah blah</div><div><br></div><div>Then, on exit it will show you what's leaking.</div><div><br></div><div>Alternatively, if you can make a script that typifies your workload (perhaps at a smaller scale) that would help to reproduce the problem.</div>
<div><br></div><div>-kb</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div lang="EN-NZ" link="blue" vlink="purple"><div><div class="im">
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"> </span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Steve</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"> </span></p>
<div class="MsoNormal" align="center" style="text-align:center"><span lang="EN-US" style="font-size:11.0pt;color:#1F497D">
<hr size="2" width="100%" align="center">
</span></div>
<p class="MsoNormal"><b><span style="font-size:11.0pt;color:#1F497D">Steve Shipway</span></b></p>
<p class="MsoNormal"><span style="font-size:10.0pt;color:#1F497D">ITS Unix Services Design Lead</span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;color:#1F497D">University of Auckland, New Zealand</span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;color:#1F497D">Floor 1, 58 Symonds Street, Auckland</span></p>
<p class="MsoNormal"><i><span style="font-size:10.0pt;color:#595959">Phone: +64 (0)9 3737599 ext 86487</span></i></p>
<p class="MsoNormal"><i><span style="font-size:10.0pt;color:#595959">DDI: +64 (0)9 924 6487</span></i></p>
<p class="MsoNormal"><i><span style="font-size:10.0pt;color:#595959">Mobile: +64 (0)21 753 189</span></i></p>
<p class="MsoNormal"><i><span style="font-size:10.0pt;color:#595959">Email: <a href="mailto:s.shipway@auckland.ac.nz" target="_blank"><span style="color:#595959">s.shipway@auckland.ac.nz</span></a></span></i></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:18.0pt;font-family:Webdings;color:green">P</span><span lang="EN-GB" style="font-size:11.0pt;color:blue"> </span><span lang="EN-GB" style="font-size:10.0pt;color:green">Please consider the environment before printing this e-mail</span><span lang="EN-GB" style="font-size:11.0pt;color:blue"> </span><span lang="EN-GB" style="font-size:7.5pt;color:navy"></span></p>
<p class="MsoNormal"><i><span style="font-size:10.0pt;color:#1F497D"> </span></i></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"> </span></p>
</div><div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt">
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:10.0pt">From:</span></b><span lang="EN-US" style="font-size:10.0pt"> kevin brintnall [mailto:<a href="mailto:kbrint@rufus.net" target="_blank">kbrint@rufus.net</a>] <br>
<b>Sent:</b> Friday, 22 October 2010 1:40 p.m.<br>
<b>To:</b> Steve Shipway<br>
<b>Cc:</b> <a href="mailto:rrd-developers@lists.oetiker.ch" target="_blank">rrd-developers@lists.oetiker.ch</a>; <a href="mailto:rrd-users@lists.oetiker.ch" target="_blank">rrd-users@lists.oetiker.ch</a><br>
<b>Subject:</b> Re: [rrd-developers] rrdcached use corrupting RRD files (trunk)</span></p>
</div>
</div><div><div></div><div class="h5">
<p class="MsoNormal"> </p>
<p class="MsoNormal">Sebastian,</p>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal">I don't think the problem is specific to rrdcached; it uses
normal librrd API. This problem likely affects any RRD access in a memory
constrained system.</p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal">Is there a lack of memory (or address space if 32-bit) on
the system? Or is it running up against per-process limits?</p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal">How does the file end up? Is it the right size?
What errors do you get (i.e. when you "rrdtool info").
What architecture are you running on? mmap() under failure
conditions is likely to be OS-specific.</p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal">What revision of trunk?</p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal">Let us know what you find re: memory leak.</p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">-kb</p>
<div>
<p class="MsoNormal">On Thu, Oct 21, 2010 at 5:07 PM, Steve Shipway <<a href="mailto:s.shipway@auckland.ac.nz" target="_blank">s.shipway@auckland.ac.nz</a>> wrote:</p>
<div>
<div>
<p class="MsoNormal">I’ve
had this happen too often now for it to be a fluke. OK, so I’m
using the trunk version of rrdtool 1.4, but (as far as I know) there is nothing
in there to modify the update code. We have a high update frequency
– approx. 20,000 MRTG targets at 5min intervals, which equates to about
70 updates per second, and it took about a week for the problem to first hit.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">It
seems that something is happening on update, possibly involving memory
allocation failure, that results in a corrupted file.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">I
have some processes that may be reading the file without using the rrdcached,
but all updates are certainly going this way (no data collection is run on this
server any more, it all comes over TCP)</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Selected
error logs show:</p>
<p class="MsoNormal">listen_thread_main:
pthread_create failed.</p>
<p class="MsoNormal">queue_thread_main:
rrd_update_r (/u01/rrdtool/maildelivery-mx1.rrd) failed with status -1.
(mmaping file '/u01/rrdtool/maildelivery-mx1.rrd': Cannot allocate memory)</p>
<p class="MsoNormal"><i>
(restarted rrdcached here)</i></p>
<p class="MsoNormal">replaying
from journal: /u01/rrdtool/journal/rrd.journal.1285603416.766523</p>
<p class="MsoNormal">Replayed
61011 entries (0 failures)</p>
<p class="MsoNormal">replaying
from journal: /u01/rrdtool/journal/rrd.journal.1285607016.766153</p>
<p class="MsoNormal">Malformed
journal entry at line 31024</p>
<p class="MsoNormal">Replayed
31023 entries (1 failures)</p>
<p class="MsoNormal">journal
processing complete</p>
<p class="MsoNormal">queue_thread_main:
rrd_update_r (/u01/rrdtool/maildelivery-mx1.rrd) failed with status -1.
('/u01/rrdtool/maildelivery-mx1.rrd' is not an RRD file)</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Although
there was only one journal failure, there were in fact several RRD files
corrupted (I suspect the ones which were open at the time of the memory
failure?) and even more with the rrd_update_r memory allocation failure.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">It
seems that the memory ran out (memory leak?) and somewhere in the rrd_update_r
something was half-done. The resultant corrupted RRD file doesn’t
even load in rrdtool, seems the header is corrupt – I don’t (yet)
understand enough of the mmap code to work out what could be causing
this. I’m also trying to track the memory usage of the rrdcached
process to see if it is indeed growing due to a leak.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">I
think there are two bugs here – first, the memory leak causing the
failure, and second, something in the code is not correctly handling a memory
allocation failure and corrupts the RRD file as a result.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Has
anyone else experienced this? And, more to the point, any RRD developers
who understand the MMAP update code want to take a look or give some pointers?</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Steve</p>
<p class="MsoNormal"> </p>
<div class="MsoNormal" align="center" style="text-align:center"><span lang="EN-US">
<hr size="2" width="100%" align="center">
</span></div>
<p class="MsoNormal"><b>Steve
Shipway</b></p>
<p class="MsoNormal"><span style="font-size:10.0pt">ITS Unix Services Design Lead</span></p>
<p class="MsoNormal"><span style="font-size:10.0pt">University of Auckland, New Zealand</span></p>
<p class="MsoNormal"><span style="font-size:10.0pt">Floor 1, 58 Symonds Street, Auckland</span></p>
<p class="MsoNormal"><i><span style="font-size:10.0pt;color:#595959">Phone: +64 (0)9 3737599 ext 86487</span></i></p>
<p class="MsoNormal"><i><span style="font-size:10.0pt;color:#595959">DDI: +64 (0)9 924 6487</span></i></p>
<p class="MsoNormal"><i><span style="font-size:10.0pt;color:#595959">Mobile: +64 (0)21 753 189</span></i></p>
<p class="MsoNormal"><i><span style="font-size:10.0pt;color:#595959">Email: <a href="mailto:s.shipway@auckland.ac.nz" target="_blank"><span style="color:#595959">s.shipway@auckland.ac.nz</span></a></span></i></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:18.0pt;font-family:Webdings;color:green">P</span><span lang="EN-GB" style="color:blue"> </span><span lang="EN-GB" style="font-size:10.0pt;color:green">Please consider the environment before printing this e-mail</span><span lang="EN-GB" style="color:blue"> </span></p>
<p class="MsoNormal"><i><span style="font-size:10.0pt"> </span></i></p>
<p class="MsoNormal"> </p>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
_______________________________________________<br>
rrd-developers mailing list<br>
<a href="mailto:rrd-developers@lists.oetiker.ch" target="_blank">rrd-developers@lists.oetiker.ch</a><br>
<a href="https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers" target="_blank">https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers</a></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
<br clear="all">
<br>
-- <br>
kevin brintnall =~ /<a href="http://kbrint@rufus.net/" target="_blank">kbrint@rufus.net/</a></p>
</div>
</div></div></div>
</div>
</div>
</blockquote></div><br><br clear="all"><br>-- <br> kevin brintnall =~ /<a href="http://kbrint@rufus.net/">kbrint@rufus.net/</a><br><br>