<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=Content-Type content="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 12 (filtered medium)">
<!--[if !mso]>
<style>
v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style>
<![endif]-->
<style>
<!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Verdana;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Webdings;
panose-1:5 3 1 2 1 5 9 6 7 3;}
@font-face
{font-family:"Arial Narrow";
panose-1:2 11 6 6 2 2 2 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=EN-NZ link=blue vlink=purple>
<div class=WordSection1>
<p class=MsoNormal>I’ve had this happen too often now for it to be a fluke.
OK, so I’m using the trunk version of rrdtool 1.4, but (as far as I know)
there is nothing in there to modify the update code. We have a high
update frequency – approx. 20,000 MRTG targets at 5min intervals, which
equates to about 70 updates per second, and it took about a week for the
problem to first hit.<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>It seems that something is happening on update, possibly involving
memory allocation failure, that results in a corrupted file.<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>I have some processes that may be reading the file without
using the rrdcached, but all updates are certainly going this way (no data
collection is run on this server any more, it all comes over TCP)<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>Selected error logs show:<o:p></o:p></p>
<p class=MsoNormal>listen_thread_main: pthread_create failed.<o:p></o:p></p>
<p class=MsoNormal>queue_thread_main: rrd_update_r (/u01/rrdtool/maildelivery-mx1.rrd)
failed with status -1. (mmaping file '/u01/rrdtool/maildelivery-mx1.rrd':
Cannot allocate memory)<o:p></o:p></p>
<p class=MsoNormal><i> (restarted rrdcached here)<o:p></o:p></i></p>
<p class=MsoNormal>replaying from journal: /u01/rrdtool/journal/rrd.journal.1285603416.766523<o:p></o:p></p>
<p class=MsoNormal>Replayed 61011 entries (0 failures)<o:p></o:p></p>
<p class=MsoNormal>replaying from journal: /u01/rrdtool/journal/rrd.journal.1285607016.766153<o:p></o:p></p>
<p class=MsoNormal>Malformed journal entry at line 31024<o:p></o:p></p>
<p class=MsoNormal>Replayed 31023 entries (1 failures)<o:p></o:p></p>
<p class=MsoNormal>journal processing complete<o:p></o:p></p>
<p class=MsoNormal>queue_thread_main: rrd_update_r
(/u01/rrdtool/maildelivery-mx1.rrd) failed with status -1.
('/u01/rrdtool/maildelivery-mx1.rrd' is not an RRD file)<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>Although there was only one journal failure, there were in
fact several RRD files corrupted (I suspect the ones which were open at the
time of the memory failure?) and even more with the rrd_update_r memory allocation
failure.<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>It seems that the memory ran out (memory leak?) and somewhere
in the rrd_update_r something was half-done. The resultant corrupted RRD
file doesn’t even load in rrdtool, seems the header is corrupt – I don’t
(yet) understand enough of the mmap code to work out what could be causing
this. I’m also trying to track the memory usage of the rrdcached
process to see if it is indeed growing due to a leak.<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>I think there are two bugs here – first, the memory
leak causing the failure, and second, something in the code is not correctly
handling a memory allocation failure and corrupts the RRD file as a result.<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>Has anyone else experienced this? And, more to the
point, any RRD developers who understand the MMAP update code want to take a
look or give some pointers?<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>Steve<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<div class=MsoNormal align=center style='text-align:center'><span lang=EN-US>
<hr size=2 width="100%" align=center>
</span></div>
<p class=MsoNormal><b>Steve Shipway<o:p></o:p></b></p>
<p class=MsoNormal><span style='font-size:10.0pt'>ITS Unix Services Design Lead<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:10.0pt'>University of Auckland, New
Zealand<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:10.0pt'>Floor 1, 58 Symonds Street,
Auckland<o:p></o:p></span></p>
<p class=MsoNormal><i><span style='font-size:10.0pt;color:#595959'>Phone: +64
(0)9 3737599 ext 86487<o:p></o:p></span></i></p>
<p class=MsoNormal><i><span style='font-size:10.0pt;color:#595959'>DDI: +64
(0)9 924 6487<o:p></o:p></span></i></p>
<p class=MsoNormal><i><span style='font-size:10.0pt;color:#595959'>Mobile: +64
(0)21 753 189<o:p></o:p></span></i></p>
<p class=MsoNormal><i><span style='font-size:10.0pt;color:#595959'>Email: <a
href="mailto:s.shipway@auckland.ac.nz"><span style='color:#595959'>s.shipway@auckland.ac.nz</span></a><o:p></o:p></span></i></p>
<p class=MsoNormal><span lang=EN-GB style='font-size:18.0pt;font-family:Webdings;
color:green'>P</span><span lang=EN-GB style='color:blue'> </span><span
lang=EN-GB style='font-size:10.0pt;font-family:"Arial Narrow","sans-serif";
color:green'>Please consider the environment before printing this e-mail</span><span
lang=EN-GB style='color:blue'> </span><span lang=EN-GB style='font-size:7.5pt;
font-family:"Verdana","sans-serif";color:navy'><o:p></o:p></span></p>
<p class=MsoNormal><i><span style='font-size:10.0pt'><o:p> </o:p></span></i></p>
<p class=MsoNormal><o:p> </o:p></p>
</div>
</body>
</html>