[rrd-users] Can I optimise this ?

[Simon Hobson]

I'm using rrdcgi (version 1.2.11-0.5 from Debian testing) to generate 
a stacked graph and accompanying legend, the source of an example is 
below. Ultimately I need to be able to handle 254 data sources (all 
IPs in a subnet) but it's taking a huge amount of memory - with 203 
sources it took 1138M of memory according to top (the machine only 
has 768M at the moment and 1G max when we get some more memory). 
Interestingly it seems to take considerably more memory for longer 
timescales, even though I have consolidated values stored :

rrdtool create ip-stats.rrd -s 300 \
   DS:total-in:DERIVE:600:0:U \
   DS:total-out:DERIVE:600:0:U \
   \
   DS:ip1-in:DERIVE:600:0:U \
   DS:ip1-out:DERIVE:600:0:U \
   DS:ip2-in:DERIVE:600:0:U \
   DS:ip2-out:DERIVE:600:0:U \
...
   DS:ip254-in:DERIVE:600:0:U \
   DS:ip254-out:DERIVE:600:0:U \
   \
   RRA:AVERAGE:0.5:1:576 \
   RRA:MAX:0.5:1:576 \
   RRA:AVERAGE:0.5:6:672 \
   RRA:MAX:0.5:6:672 \
   RRA:AVERAGE:0.5:24:732 \
   RRA:MAX:0.5:24:732 \
   RRA:AVERAGE:0.5:144:1460 \
   RRA:MAX:0.5:144:1460

# CFs for :
#   1 x 576    48hrx 5m
#   6 x 672    14d x 1/2hr
#  24 x 732    61d x 2hr
# 144 x 1460  730d x 12hr




The preamble
         <RRD::GRAPH /var/www/graphs/gr3det-off-366d.png
         --title="Group3 - Last Year" -v "bytes/second"
         --end now --start end-366d
         --lazy
         --width 600 --height 300
         --imginfo '<IMG SRC="../graphs/%s" width=%lu height=%lu 
ALT="Graph data for of Group3 over Last Year">'

Then the data sources
         DEF:1-in=/var/rrd/ip-stats.rrd:ip1-in:AVERAGE
         DEF:1-out=/var/rrd/ip-stats.rrd:ip1-out:AVERAGE
         CDEF:i1-out=1-out,-1,*
         VDEF:v1-in=1-in,AVERAGE
         VDEF:v1-inmax=1-in,MAXIMUM
         VDEF:v1-out=1-out,AVERAGE
         VDEF:v1-outmax=1-out,MAXIMUM

This is for IP a.b.c.1, the 1-in and i1-out are used for plotting the 
graph, v1-* are used for the legend (avg & max, in & out). The block 
is repeated for each address selected to be graphed.




Get the sum of the rates (ie a total)
         CDEF:datainavg=1-in,2-in,+,3-in,+
         VDEF:vdatainavg=datainavg,AVERAGE
         CDEF:dataoutavg=1-out,2-out,+,3-out,+
         VDEF:vdataoutavg=dataoutavg,AVERAGE

This is the bit I think should be capable of significant 
optimisation. I already have the average in & out for each dataset, 
so I thought I should be able to just do :

VDEF:vdatainavg=v1-in, v2-in,+,v3-in,+

but this results in "[ERROR: Cannot parse vname from 
'VDEF:VDEF:vdatainavg=v1-in,']"

Looking again I can see that VDEF is looking to process an array and 
consolidate it to a single value - ie there nothing that will do 
basic arithmetic on single values ? It seems daft to add loads of 
arrays together when I only want a sum of single values and I already 
have the values.



Finally, plot the data :
         COMMENT:"  IP Address        In avg       In max 
Out avg      Out max\n"

         AREA:1-in#FF7F7F:"195.8.169.1  "
         GPRINT:v1-in:"%6.2lf %sBps"
         GPRINT:v1-inmax:"%6.2lf %sBps"
         COMMENT:" "
         GPRINT:v1-out:"%6.2lf %sBps"
         GPRINT:v1-outmax:"%6.2lf %sBps"
         COMMENT:" something.somedomain.net\n"

         AREA:2-in#7FFF7F:"195.8.169.2  ":STACK
         GPRINT:v2-in:"%6.2lf %sBps"
         ...

and the outbound stack :
         AREA:i1-out#FF7F7F:
         AREA:i2-out#7FFF7F::STACK
         AREA:i3-out#7F7FFF::STACK


and the totals :
         COMMENT:"\n"
         COMMENT:"In  avg"
         GPRINT:vdatainavg:"%6.2lf %sBps\n"

         COMMENT:"Out avg"
         GPRINT:vdataoutavg:"%6.2lf %sBps"

         COMMENT:"  Data to <RRD::TIME::LAST /var/rrd/ip-stats.rrd '%a 
%b %d %Y  %H\:%M\:%S %Z'>\n"

--
Unsubscribe mailto:rrd-users-request at list.ee.ethz.ch?subject=unsubscribe
Help        mailto:rrd-users-request at list.ee.ethz.ch?subject=help
Archive     http://lists.ee.ethz.ch/rrd-users
WebAdmin    http://lists.ee.ethz.ch/lsg2.cgi