<html><body><div style="color:#000; background-color:#fff; font-family:arial, helvetica, sans-serif;font-size:10pt"><div style="font-family: arial, helvetica, sans-serif; font-size: 10pt; "><span><br></span></div><div style="font-family: arial, helvetica, sans-serif; font-size: 10pt; "><span>Tobi,</span></div><div style="font-family: arial, helvetica, sans-serif; font-size: 10pt; "><span><br></span></div><div style="font-family: arial, helvetica, sans-serif; font-size: 10pt; ">I'm curious what benchmarks/tracing you do during development to localize performance bottlenecks in rrdtool graphing.</div><div style="font-family: arial, helvetica, sans-serif; font-size: 10pt; "><br></div><div style="font-family: arial, helvetica, sans-serif; font-size: 10pt; ">The legend does slow graph generation (23ms for a standard graph no-legend for me, 34ms with the legend added) -- worse on long legend graphs; which is one reason I implemented legends in datatables (via
JSON) for sizable legend lengths (50 +) instead of having rrdtool draw the legend ( plus of course datatables is a better format for that sort of thing, but I digress. )</div><div style="font-family: arial, helvetica, sans-serif; font-size: 10pt; "><br></div><div style="font-family: arial, helvetica, sans-serif; font-size: 10pt; ">What I'm really curious about is this during a graph/xport creation of 25,000 (25K) datasources rrdtool (perl RRDs) spends considerable time doing this:</div><div style="font-family: arial, helvetica, sans-serif; font-size: 10pt; "><br></div><div><div><font size="2">mremap(0x7fe0454df000, 56258560, 56262656, MREMAP_MAYMOVE) = 0x7fe0454df000</font></div><div><font size="2">mremap(0x7fe0454df000, 56262656, 56266752, MREMAP_MAYMOVE) = 0x7fe0454df000</font></div><div><font size="2">mremap(0x7fe0454df000, 56266752, 56270848, MREMAP_MAYMOVE) = 0x7fe0454df000</font></div><div><font size="2">mremap(0x7fe0454df000, 56270848, 56274944,
MREMAP_MAYMOVE) = 0x7fe0454df000</font></div><div><font size="2">mremap(0x7fe0454df000, 56274944, 56279040, MREMAP_MAYMOVE) = 0x7fe0454df000</font></div><div><font size="2">mremap(0x7fe0454df000, 56279040, 56283136, MREMAP_MAYMOVE) = 0x7fe0454df000</font></div><div><font size="2">mremap(0x7fe0454df000, 56283136, 56287232, MREMAP_MAYMOVE) = 0x7fe0454df000</font></div><div><font size="2">mremap(0x7fe0454df000, 56287232, 56291328, MREMAP_MAYMOVE) = 0x7fe0454df000</font></div><div><font size="2">mremap(0x7fe0454df000, 56291328, 56295424, MREMAP_MAYMOVE) = 0x7fe0454df000</font></div><div><font size="2">mremap(0x7fe0454df000, 56295424, 56299520, MREMAP_MAYMOVE) = 0x7fe0454df000</font></div><div><font size="2">mremap(0x7fe0454df000, 56299520, 56303616, MREMAP_MAYMOVE) = 0x7fe0454df000</font></div><div style="font-family: arial, helvetica, sans-serif; font-size: 10pt; "><br></div></div><div style="font-family: arial, helvetica, sans-serif; font-size: 10pt;
"><br></div><div style="font-family: arial, helvetica, sans-serif; font-size: 10pt; ">... granted the performance of what it's doing is still very good (99 seconds for the graph on 25k datasources for 3 hours of 1 minute data == ~45k datapoints calculated <span style="font-size: 10pt; ">per second), it certainly seems to have substantially decreasing performance (more than linear) as datasources increase for a request. (note: I say very good in comparison to other numerical data export systems.)</span></div><div style="font-family: arial, helvetica, sans-serif; font-size: 10pt; "><br></div><div style="font-family: arial, helvetica, sans-serif; font-size: 10pt; ">Other tests (less datasources longer time frame) have datapoints per second in the millions:</div><div style="font-family: arial, helvetica, sans-serif; font-size: 10pt; "><br></div><div style="font-family: arial, helvetica, sans-serif; font-size: 10pt; "><span class="Apple-tab-span"
style="white-space:pre"> </span>5,000 datasources same time-frame/data takes 5.48 seconds == ~164k datapoints calculated <span style="font-size: 10pt; ">per second.</span></div><div style="font-family: arial, helvetica, sans-serif; font-size: 10pt; "><span class="Apple-tab-span" style="white-space:pre"> </span>100 datasources 90 day time-frame 5 minute data, 1.32 seconds == 1.9 million datapoints calculated per second.</div><div style="font-family: arial, helvetica, sans-serif; font-size: 10pt; "><br></div><div style="font-family: arial, helvetica, sans-serif; font-size: 10pt; ">Oddly enough the above mremap()'s happen before files are even opened and read. I've strace'd httpd during a request like this and it spends all it's time doing mremap's then very quickly reads all datasources and produces the graph... more than a minute mremap'ing then seconds open/read/close'ing data files and producing the graph.</div><div style="font-family:
arial, helvetica, sans-serif; font-size: 10pt; "><br></div><div style="font-family: arial, helvetica, sans-serif; font-size: 10pt; ">I mention it happens with xports because I have the dual purpose of requiring the above type of merge (it's sum() and rpn math for many datasources) to be quick.</div><div style="font-family: arial, helvetica, sans-serif; font-size: 10pt; "><br></div><div style="font-family: arial, helvetica, sans-serif; font-size: 10pt; ">It's as if the rrdtool syntax parser is very slow or rrdtool spends a lot of time up front creating a 'workspace' in memory to calculate with and constantly resizes memory?</div><div style="font-family: arial, helvetica, sans-serif; font-size: 10pt; "><br></div><div style="font-family: arial, helvetica, sans-serif; font-size: 10pt; ">Any tips on how to trace down why the above occurs so I can work-on/suggest a fix would be helpful.</div><div style="font-family: arial, helvetica, sans-serif; font-size:
10pt; "><br></div><div style="font-family: arial, helvetica, sans-serif; font-size: 10pt; ">Thanks,</div><div style="font-family: arial, helvetica, sans-serif; font-size: 10pt; ">-Ryan</div><div style="font-family: arial, helvetica, sans-serif; font-size: 10pt; "><br></div> <div style="font-family: arial, helvetica, sans-serif; font-size: 10pt; "> <div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; "> <div dir="ltr"> <font size="2" face="Arial"> <hr size="1"> <b><span style="font-weight:bold;">From:</span></b> Tobias Oetiker <tobi@oetiker.ch><br> <b><span style="font-weight: bold;">To:</span></b> Christoph Anton Mitterer <calestyo@scientia.net> <br><b><span style="font-weight: bold;">Cc:</span></b> rrd-users@lists.oetiker.ch <br> <b><span style="font-weight: bold;">Sent:</span></b> Sunday, August 12, 2012 12:07 AM<br> <b><span style="font-weight: bold;">Subject:</span></b> Re: [rrd-users] RRA/RRD
tuning<br> </font> </div> <br>... snip...</div><div style="font-family: times new roman, new york, times, serif; font-size: 12pt;" class="yui_3_2_0_23_134475876287263"><br>the biggest win for graphing time would be to integrate support for<br>setting the png compression level into rrdtool ... libcairo sets<br>this very high, and thus spends considerable time compressing the<br>resulting png file ... setting the compression level to 1 should<br>give a considerable performance gain ...<br><br>I will be glad to integrate your patch<br><br>cheers<br>tobi<br>><br>> Thanks for your insights,<br>> Chris.<br>><br><br>-- <br>Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland<br>http://it.oetiker.ch <a ymailto="mailto:tobi@oetiker.ch" href="mailto:tobi@oetiker.ch">tobi@oetiker.ch</a> ++41 62 775 9902 / sb: -9900<br><br>_______________________________________________<br>rrd-users mailing list<br><a
ymailto="mailto:rrd-users@lists.oetiker.ch" href="mailto:rrd-users@lists.oetiker.ch">rrd-users@lists.oetiker.ch</a><br><a href="https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users" target="_blank">https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users</a><br><br><br> </div> </div> </div></body></html>