<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns:p="urn:schemas-microsoft-com:office:powerpoint" xmlns:a="urn:schemas-microsoft-com:office:access" xmlns:dt="uuid:C2F41010-65B3-11d1-A29F-00AA00C14882" xmlns:s="uuid:BDC6E3F0-6DA3-11d1-A2A3-00AA00C14882" xmlns:rs="urn:schemas-microsoft-com:rowset" xmlns:z="#RowsetSchema" xmlns:b="urn:schemas-microsoft-com:office:publisher" xmlns:ss="urn:schemas-microsoft-com:office:spreadsheet" xmlns:c="urn:schemas-microsoft-com:office:component:spreadsheet" xmlns:odc="urn:schemas-microsoft-com:office:odc" xmlns:oa="urn:schemas-microsoft-com:office:activation" xmlns:html="http://www.w3.org/TR/REC-html40" xmlns:q="http://schemas.xmlsoap.org/soap/envelope/" xmlns:rtc="http://microsoft.com/officenet/conferencing" xmlns:D="DAV:" xmlns:Repl="http://schemas.microsoft.com/repl/" xmlns:mt="http://schemas.microsoft.com/sharepoint/soap/meetings/" xmlns:x2="http://schemas.microsoft.com/office/excel/2003/xml" xmlns:ppda="http://www.passport.com/NameSpace.xsd" xmlns:ois="http://schemas.microsoft.com/sharepoint/soap/ois/" xmlns:dir="http://schemas.microsoft.com/sharepoint/soap/directory/" xmlns:ds="http://www.w3.org/2000/09/xmldsig#" xmlns:dsp="http://schemas.microsoft.com/sharepoint/dsp" xmlns:udc="http://schemas.microsoft.com/data/udc" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:sub="http://schemas.microsoft.com/sharepoint/soap/2002/1/alerts/" xmlns:ec="http://www.w3.org/2001/04/xmlenc#" xmlns:sp="http://schemas.microsoft.com/sharepoint/" xmlns:sps="http://schemas.microsoft.com/sharepoint/soap/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:udcs="http://schemas.microsoft.com/data/udc/soap" xmlns:udcxf="http://schemas.microsoft.com/data/udc/xmlfile" xmlns:udcp2p="http://schemas.microsoft.com/data/udc/parttopart" xmlns:wf="http://schemas.microsoft.com/sharepoint/soap/workflow/" xmlns:dsss="http://schemas.microsoft.com/office/2006/digsig-setup" xmlns:dssi="http://schemas.microsoft.com/office/2006/digsig" xmlns:mdssi="http://schemas.openxmlformats.org/package/2006/digital-signature" xmlns:mver="http://schemas.openxmlformats.org/markup-compatibility/2006" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns:mrels="http://schemas.openxmlformats.org/package/2006/relationships" xmlns:spwp="http://microsoft.com/sharepoint/webpartpages" xmlns:ex12t="http://schemas.microsoft.com/exchange/services/2006/types" xmlns:ex12m="http://schemas.microsoft.com/exchange/services/2006/messages" xmlns:pptsl="http://schemas.microsoft.com/sharepoint/soap/SlideLibrary/" xmlns:spsl="http://microsoft.com/webservices/SharePointPortalServer/PublishedLinksService" xmlns:Z="urn:schemas-microsoft-com:" xmlns:st="" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=Content-Type content="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 12 (filtered medium)">
<!--[if !mso]>
<style>
v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style>
<![endif]-->
<style>
<!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Verdana;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Webdings;
panose-1:5 3 1 2 1 5 9 6 7 3;}
@font-face
{font-family:"Arial Narrow";
panose-1:2 11 6 6 2 2 2 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:845021897;
mso-list-type:hybrid;
mso-list-template-ids:-395416008 336134159 336134169 336134171 336134159 336134169 336134171 336134159 336134169 336134171;}
@list l0:level1
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=EN-NZ link=blue vlink=purple>
<div class=WordSection1>
<p class=MsoNormal>Here are the preliminary results for my performance testing
of MRTG/RRD. It is very interesting that some things which I had thought
to be significant turned out not to be, but others (such as daemon mode) were
much more important. As a result of this investigation, I expect to be
making serious changes to our organisation’s configuration in order to
improve performance.<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>All tests were done with a dual 2.5GHz Xeon system, 4GB memory,
RedHat Enterprise Linux 5.4.<br>
3164 SNMP Targets over 32 devices, with a clean pass (no SNMP errors).<o:p></o:p></p>
<p class=MsoNormal>In all cases, a clean directory was used, and the RRD files
created by an initial pass before running the test.<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal><b>Overall observations<o:p></o:p></b></p>
<p class=MsoNormal>Strangely, running RTG in Daemon mode (as opposed to scheduled
mode) moves the CPU usage from ‘Nice’ to ‘User’
classification.<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal><b>Native mode vs. RRD mode <a
href="http://oss.oetiker.ch/mrtg/doc/mrtg-rrd.en.html">http://oss.oetiker.ch/mrtg/doc/mrtg-rrd.en.html</a>
<o:p></o:p></b></p>
<p class=MsoNormal>As expected, running in RRD mode is much more
efficient. In RRD mode, you use only 35% of the CPU requirements,
although disk IO is not greatly changed.<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal><b>RRDTool Versions<o:p></o:p></b></p>
<p class=MsoNormal>There is no significant change in the CPU usage between
RRDTool 1.2.12, 1.3.9 and 1.4.4. RRDTool 1.4.4 seems to require
slightly more disk I/O when run in Scheduled mode, but <i>less</i> when run in
Daemon mode – probably due to its more efficient memory mapped I/O.<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal><b>MRTG Daemon mode<o:p></o:p></b></p>
<p class=MsoNormal>Running MRTG in daemon mode allows it to keep file
descriptors open, cache the configuration file contents, and so on. The
performance gain seems to be consistently in the region of 20%, mo matter which
version of RRDTool you are using.<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal><b>RRDTool Caching Daemon <a
href="http://oss.oetiker.ch/rrdtool/doc/rrdcached.en.html">http://oss.oetiker.ch/rrdtool/doc/rrdcached.en.html</a>
<o:p></o:p></b></p>
<p class=MsoNormal>The rrdcached was introduced with RRDTool 1.4, and allows a
separate daemon to take control of disk writes. For testing, this was
left with default values, but if configured with more aggressive (and risky)
caching you could make serious gains in I/O.<o:p></o:p></p>
<p class=MsoNormal>This seems to have very little impact on overall CPU use -- remember
we’re taking into account the requirements of the caching daemon in
addition to those of MRTG – although you could run the rrdcached process on
a separate server to spread the load or achieve a distributed setup. <o:p></o:p></p>
<p class=MsoNormal>However, the gains in disk IO are significant, particularly
if you take into account that any web front-end can also benefit from accessing
the cache rather than the data files. I was experiencing a saving of 60%+
in disk writes, although this will probably lessen if a much larger sample
window is taken (I’ll be running more tests on this in the future).<o:p></o:p></p>
<p class=MsoNormal>To make MRTG use the rrdcached, you need to set the
RRDCACHED_ADDRESS environment variable before starting.<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal><b>FIlesystem mount options<o:p></o:p></b></p>
<p class=MsoNormal>Still to be tested but not anticipated to have any
significant impact<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal><b>Preliminary Recommendations<o:p></o:p></b></p>
<p class=MsoNormal>If performance is your primary goal, then the best course of
action appears to be:<o:p></o:p></p>
<p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l0 level1 lfo1'><![if !supportLists]><span
style='mso-list:Ignore'>1.<span style='font:7.0pt "Times New Roman"'>
</span></span><![endif]>Use RRDTool, not Native mode for MRTG<o:p></o:p></p>
<p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l0 level1 lfo1'><![if !supportLists]><span
style='mso-list:Ignore'>2.<span style='font:7.0pt "Times New Roman"'>
</span></span><![endif]>Run MRTG in daemon mode rather than via cron or other
scheduler<o:p></o:p></p>
<p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l0 level1 lfo1'><![if !supportLists]><span
style='mso-list:Ignore'>3.<span style='font:7.0pt "Times New Roman"'>
</span></span><![endif]>Use RRDTool 1.4.x in order to benefit from enhanced
disk IO routines<o:p></o:p></p>
<p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l0 level1 lfo1'><![if !supportLists]><span
style='mso-list:Ignore'>4.<span style='font:7.0pt "Times New Roman"'>
</span></span><![endif]>Use the rrdcached, with as large a timeout (-w) as you
dare (default is 5min). Set the delay (-z) if you experience disk IO
going in bursts. Increase the threads (-t) if you have fast disk (default
4).<o:p></o:p></p>
<p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l0 level1 lfo1'><![if !supportLists]><span
style='mso-list:Ignore'>5.<span style='font:7.0pt "Times New Roman"'>
</span></span><![endif]>Consider running multiple MRTG processes on separate
servers, with the rrdcached process (and web frontend) on a separate server.
Since the majority of the load comes from MRTG this allows more processes
before the disk IO is saturated.<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>I will be doing more investigation on a distributed MRTG
setup using the rrdcached, which I hope to be able to demonstrate at LISA10
this year. I will also be adding a section to the MRTG book on the new
rrdcached features, once I’ve checked them through thoroughly.<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>All feedback welcome…<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>Steve<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<div class=MsoNormal align=center style='text-align:center'><span lang=EN-US>
<hr size=2 width="100%" align=center>
</span></div>
<p class=MsoNormal><b>Steve Shipway<o:p></o:p></b></p>
<p class=MsoNormal><span style='font-size:10.0pt'>ITS Unix Services Design Lead<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:10.0pt'>University of Auckland<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:10.0pt'>Floor 2, 58 Symonds Street<o:p></o:p></span></p>
<p class=MsoNormal><i><span style='font-size:10.0pt'>09 3737599 ext 86487<o:p></o:p></span></i></p>
<p class=MsoNormal><span lang=EN-GB style='font-size:18.0pt;font-family:Webdings;
color:green'>P</span><span lang=EN-GB style='color:blue'> </span><span
lang=EN-GB style='font-size:10.0pt;font-family:"Arial Narrow","sans-serif";
color:green'>Please consider the environment before printing this e-mail</span><span
lang=EN-GB style='color:blue'> </span><span lang=EN-GB style='font-size:7.5pt;
font-family:"Verdana","sans-serif";color:navy'><o:p></o:p></span></p>
<p class=MsoNormal><i><span style='font-size:10.0pt'><o:p> </o:p></span></i></p>
<p class=MsoNormal><o:p> </o:p></p>
</div>
</body>
</html>