<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=Content-Type content="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 12 (filtered medium)">
<!--[if !mso]>
<style>
v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style>
<![endif]-->
<style>
<!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Verdana;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Webdings;
panose-1:5 3 1 2 1 5 9 6 7 3;}
@font-face
{font-family:"Arial Narrow";
panose-1:2 11 6 6 2 2 2 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=EN-NZ link=blue vlink=purple>
<div class=WordSection1>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>The corrupted file ends up the correct size; however the entire
file is filled with zeroes (fortunately, we archive our RRD files nightly so I
can go back and retrieve the last uncorrupted version plus the corrupted
version)<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>The system is not (normally) memory or process-constrained;
there is in fact nothing to speak of running apart from apache and the
rrdcached daemon. The rrdinfo response is ‘not an RRD file’,
since it doesn’t have the RRD header.<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>It has run fine for a whole week at these rates before the
problem hit; so that’s why I think it might be a leak in the RRD
functions (which would of course not show up in a non-daemon situation).
We use the remote update, info and (occasionally) create via the TCP socket;
plus the info, last, flush and fetch via the UNIX socket.<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>The build is the absolute latest r2136 .<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>The memory usage of the rrdcached process is definitely
increasing; however that may also be due to the number of items in the
queue? It is currently at 768m virtual, 560m physical (17% usage) which
seems somewhat high to me, even for 20,000+ RRD files. Eventually it will
hit address-space limits (this is a 32bit RHEL5 box with 4G physical memory)<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>Unfortunately I don’t have any of the nice developer tools
for tracking memory leaks…<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>Steve<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<div class=MsoNormal align=center style='text-align:center'><span lang=EN-US
style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>
<hr size=2 width="100%" align=center>
</span></div>
<p class=MsoNormal><b><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>Steve Shipway<o:p></o:p></span></b></p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>ITS Unix Services Design Lead<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>University of Auckland, New Zealand<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>Floor 1, 58 Symonds Street, Auckland<o:p></o:p></span></p>
<p class=MsoNormal><i><span style='font-size:10.0pt;font-family:"Calibri","sans-serif";
color:#595959'>Phone: +64 (0)9 3737599 ext 86487<o:p></o:p></span></i></p>
<p class=MsoNormal><i><span style='font-size:10.0pt;font-family:"Calibri","sans-serif";
color:#595959'>DDI: +64 (0)9 924 6487<o:p></o:p></span></i></p>
<p class=MsoNormal><i><span style='font-size:10.0pt;font-family:"Calibri","sans-serif";
color:#595959'>Mobile: +64 (0)21 753 189<o:p></o:p></span></i></p>
<p class=MsoNormal><i><span style='font-size:10.0pt;font-family:"Calibri","sans-serif";
color:#595959'>Email: <a href="mailto:s.shipway@auckland.ac.nz"><span
style='color:#595959'>s.shipway@auckland.ac.nz</span></a><o:p></o:p></span></i></p>
<p class=MsoNormal><span lang=EN-GB style='font-size:18.0pt;font-family:Webdings;
color:green'>P</span><span lang=EN-GB style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:blue'> </span><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial Narrow","sans-serif";
color:green'>Please consider the environment before printing this e-mail</span><span
lang=EN-GB style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:blue'> </span><span lang=EN-GB style='font-size:7.5pt;font-family:"Verdana","sans-serif";
color:navy'><o:p></o:p></span></p>
<p class=MsoNormal><i><span style='font-size:10.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></i></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<div style='border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt'>
<div>
<div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm'>
<p class=MsoNormal><b><span lang=EN-US style='font-size:10.0pt;font-family:
"Tahoma","sans-serif"'>From:</span></b><span lang=EN-US style='font-size:10.0pt;
font-family:"Tahoma","sans-serif"'> kevin brintnall [mailto:kbrint@rufus.net] <br>
<b>Sent:</b> Friday, 22 October 2010 1:40 p.m.<br>
<b>To:</b> Steve Shipway<br>
<b>Cc:</b> rrd-developers@lists.oetiker.ch; rrd-users@lists.oetiker.ch<br>
<b>Subject:</b> Re: [rrd-developers] rrdcached use corrupting RRD files (trunk)<o:p></o:p></span></p>
</div>
</div>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>Sebastian,<o:p></o:p></p>
<div>
<p class=MsoNormal><o:p> </o:p></p>
</div>
<div>
<p class=MsoNormal>I don't think the problem is specific to rrdcached; it uses
normal librrd API. This problem likely affects any RRD access in a memory
constrained system.<o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><o:p> </o:p></p>
</div>
<div>
<p class=MsoNormal>Is there a lack of memory (or address space if 32-bit) on
the system? Or is it running up against per-process limits?<o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><o:p> </o:p></p>
</div>
<div>
<p class=MsoNormal>How does the file end up? Is it the right size?
What errors do you get (i.e. when you "rrdtool info").
What architecture are you running on? mmap() under failure
conditions is likely to be OS-specific.<o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><o:p> </o:p></p>
</div>
<div>
<p class=MsoNormal>What revision of trunk?<o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><o:p> </o:p></p>
</div>
<div>
<p class=MsoNormal>Let us know what you find re: memory leak.<o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><o:p> </o:p></p>
</div>
<div>
<p class=MsoNormal style='margin-bottom:12.0pt'>-kb<o:p></o:p></p>
<div>
<p class=MsoNormal>On Thu, Oct 21, 2010 at 5:07 PM, Steve Shipway <<a
href="mailto:s.shipway@auckland.ac.nz">s.shipway@auckland.ac.nz</a>> wrote:<o:p></o:p></p>
<div>
<div>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>I’ve
had this happen too often now for it to be a fluke. OK, so I’m
using the trunk version of rrdtool 1.4, but (as far as I know) there is nothing
in there to modify the update code. We have a high update frequency
– approx. 20,000 MRTG targets at 5min intervals, which equates to about
70 updates per second, and it took about a week for the problem to first hit.<o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>It
seems that something is happening on update, possibly involving memory
allocation failure, that results in a corrupted file.<o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>I
have some processes that may be reading the file without using the rrdcached,
but all updates are certainly going this way (no data collection is run on this
server any more, it all comes over TCP)<o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Selected
error logs show:<o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>listen_thread_main:
pthread_create failed.<o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>queue_thread_main:
rrd_update_r (/u01/rrdtool/maildelivery-mx1.rrd) failed with status -1.
(mmaping file '/u01/rrdtool/maildelivery-mx1.rrd': Cannot allocate memory)<o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><i>
(restarted rrdcached here)</i><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>replaying
from journal: /u01/rrdtool/journal/rrd.journal.1285603416.766523<o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Replayed
61011 entries (0 failures)<o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>replaying
from journal: /u01/rrdtool/journal/rrd.journal.1285607016.766153<o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Malformed
journal entry at line 31024<o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Replayed
31023 entries (1 failures)<o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>journal
processing complete<o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>queue_thread_main:
rrd_update_r (/u01/rrdtool/maildelivery-mx1.rrd) failed with status -1.
('/u01/rrdtool/maildelivery-mx1.rrd' is not an RRD file)<o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Although
there was only one journal failure, there were in fact several RRD files
corrupted (I suspect the ones which were open at the time of the memory
failure?) and even more with the rrd_update_r memory allocation failure.<o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>It
seems that the memory ran out (memory leak?) and somewhere in the rrd_update_r
something was half-done. The resultant corrupted RRD file doesn’t
even load in rrdtool, seems the header is corrupt – I don’t (yet)
understand enough of the mmap code to work out what could be causing
this. I’m also trying to track the memory usage of the rrdcached
process to see if it is indeed growing due to a leak.<o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>I
think there are two bugs here – first, the memory leak causing the
failure, and second, something in the code is not correctly handling a memory
allocation failure and corrupts the RRD file as a result.<o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Has
anyone else experienced this? And, more to the point, any RRD developers
who understand the MMAP update code want to take a look or give some pointers?<o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Steve<o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p>
<div class=MsoNormal align=center style='text-align:center'><span lang=EN-US>
<hr size=2 width="100%" align=center>
</span></div>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><b>Steve
Shipway</b><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span
style='font-size:10.0pt'>ITS Unix Services Design Lead</span><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span
style='font-size:10.0pt'>University of Auckland, New Zealand</span><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span
style='font-size:10.0pt'>Floor 1, 58 Symonds Street, Auckland</span><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><i><span
style='font-size:10.0pt;color:#595959'>Phone: +64 (0)9 3737599 ext 86487</span></i><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><i><span
style='font-size:10.0pt;color:#595959'>DDI: +64 (0)9 924 6487</span></i><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><i><span
style='font-size:10.0pt;color:#595959'>Mobile: +64 (0)21 753 189</span></i><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><i><span
style='font-size:10.0pt;color:#595959'>Email: <a
href="mailto:s.shipway@auckland.ac.nz" target="_blank"><span style='color:#595959'>s.shipway@auckland.ac.nz</span></a></span></i><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span
lang=EN-GB style='font-size:18.0pt;font-family:Webdings;color:green'>P</span><span
lang=EN-GB style='color:blue'> </span><span lang=EN-GB style='font-size:10.0pt;
color:green'>Please consider the environment before printing this e-mail</span><span
lang=EN-GB style='color:blue'> </span><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><i><span
style='font-size:10.0pt'> </span></i><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p>
</div>
</div>
<p class=MsoNormal style='margin-bottom:12.0pt'><br>
_______________________________________________<br>
rrd-developers mailing list<br>
<a href="mailto:rrd-developers@lists.oetiker.ch">rrd-developers@lists.oetiker.ch</a><br>
<a href="https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers"
target="_blank">https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers</a><o:p></o:p></p>
</div>
<p class=MsoNormal style='margin-bottom:12.0pt'><br>
<br clear=all>
<br>
-- <br>
kevin brintnall =~ /<a href="http://kbrint@rufus.net/">kbrint@rufus.net/</a><o:p></o:p></p>
</div>
</div>
</div>
</body>
</html>