<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 12 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 70.85pt 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="ET" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Hi<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I’ve been trying for 3 days to migrate our Munin server from 32bit to 64bit instance (both are VM-s with pretty much same settings and OS – SLES11).<o:p></o:p></p>
<p class="MsoNormal">Problem is that it simply does not perform – it can’t update all PNG files in 5 minute timeframe (was perfectly OK in 32bit instance).<o:p></o:p></p>
<p class="MsoNormal">I’ve traced it down to rrdtool and it’s lazy option.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I’ve made a simple test.sh that I took from output of munin debug, it’s about 37K and it consists of one big „rrdtool graph“ command, which generates PNG of yearly irqstats graph of one server (because seems that irqstats graphs have the
most source data and take the most time to calculate)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">When I delete yearly PNG file in the OLD (32bit) server and run test.sh, then first run takes pretty long time but all the consecutive times it takes only ~0.06s<o:p></o:p></p>
<p class="MsoNormal">When I delete yearly PNG file in the NEW (64bit) server and run test.sh, then first run takes ~1.2s but all the consecutive times it takes 0.6-0.8s, which is more than 10 times longer than in the old server! Timestamp of the PNG file remains
the same each time, its does not get updated.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I run strace on both servers to see what’s going on, here are the excerpts of where it starts to differ:<o:p></o:p></p>
<p class="MsoNormal">Old server:<o:p></o:p></p>
<p class="MsoNormal">[pid 13016] mremap(0xb6a7f000, 1236992, 1241088, MREMAP_MAYMOVE) = 0xb6a7f000<o:p></o:p></p>
<p class="MsoNormal">[pid 13016] stat64("/srv/www/vhosts/munin.sise/kisise/myyr.kisise/irqstats-year.png", {st_mode=S_IFREG|0644, st_size=82285, ...}) = 0<o:p></o:p></p>
<p class="MsoNormal">[pid 13016] time(NULL) = 1385553085<o:p></o:p></p>
<p class="MsoNormal">[pid 13016] open("/srv/www/vhosts/munin.sise/kisise/myyr.kisise/irqstats-year.png", O_RDONLY) = 3<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">After that it pretty much finishes (doesn’t touch any .rrd file).<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Same location in new server:<o:p></o:p></p>
<p class="MsoNormal">[pid 28160] mremap(0x7f2d188ed000, 1302528, 1306624, MREMAP_MAYMOVE) = 0x7f2d188ed000<o:p></o:p></p>
<p class="MsoNormal">[pid 28160] open("/usr/lib/locale/\2/LC_NUMERIC", O_RDONLY) = -1 ENOENT (No such file or directory)<o:p></o:p></p>
<p class="MsoNormal">[pid 28160] stat("/srv/www/vhosts/munin.sise/kisise/myyr.kisise/irqstats-year.png", {st_mode=S_IFREG|0644, st_size=43415, ...}) = 0<o:p></o:p></p>
<p class="MsoNormal">[pid 28160] open("/srv/www/vhosts/munin.sise/kisise/myyr.kisise/irqstats-year.png", O_RDONLY) = 3<o:p></o:p></p>
<p class="MsoNormal">[pid 28160] fstat(3, {st_mode=S_IFREG|0644, st_size=43415, ...}) = 0<o:p></o:p></p>
<p class="MsoNormal">[pid 28160] mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f2d1cab5000<o:p></o:p></p>
<p class="MsoNormal">[pid 28160] read(3, "\211PNG\r\n\32\n\0\0\0\rIHDR\0\0\1\361\0\0\4\253\10\6\0\0\0\202\37\201"..., 4096) = 4096<o:p></o:p></p>
<p class="MsoNormal">[pid 28160] close(3) = 0<o:p></o:p></p>
<p class="MsoNormal">[pid 28160] munmap(0x7f2d1cab5000, 4096) = 0<o:p></o:p></p>
<p class="MsoNormal">[pid 28160] open("/mnt/data/munin/kisise/myyr.kisise-irqstats-iMIS-d.rrd", O_RDONLY) = 3<o:p></o:p></p>
<p class="MsoNormal">[pid 28160] fstat(3, {st_mode=S_IFREG|0644, st_size=2765800, ...}) = 0<o:p></o:p></p>
<p class="MsoNormal">[pid 28160] fadvise64(3, 0, 0, POSIX_FADV_RANDOM) = 0<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">And after that it starts to scan every single (about 100 of them) irqstats .rrd file of that server (which is the reason that it takes so much time).<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Seems that on older server it notices (like it should?) that yearly PNG file is fresh enough and does not try to recreate it but newer server stupidly tries to recalculate it?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">test.sh in both servers are exactly the same, here’s the head:<o:p></o:p></p>
<p class="MsoNormal">rrdtool 'graph' '--font' 'DEFAULT:0:DejaVuSans' '--font' 'LEGEND:7:DejaVuSansMono' '-W' 'Munin 1.4.6' /srv/www/vhosts/munin.sise/kisise/myyr.kisise/irqstats-year.png' '--title' 'Individual interrupts - by year' '--start' '-400d'
'--base' '1000' '-l' '0;' '--vertical-label' 'interrupts / second' '--height' '250' '--width' '400' '--imgformat' 'PNG' '--lazy' \ yada-yada-yada...<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">rrdtool version in both servers is 1.3.7 (was 1.4.7 in new server but I compiled and installed older one to make sure it’s not because of that).<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Heeeeelp!<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Best regards,<o:p></o:p></p>
<p class="MsoNormal">Paavo<o:p></o:p></p>
</div>
</body>
</html>