[rrd-users] RRDTool fails in pipe mode (stdin mode) with large input stream (somewhere around 10k)

Fri May 27 05:07:06 MEST 2005

Greetings,

The basic gist of the error I'm experiencing is that rrdtool, when
called in pipe/stdin mode with a very large input to summarize a bunch
of graphs, seems to attempt to parse the single command as multiple
commands, getting confused somewhere around the 10k mark.

Here's some background information on my system:

FreeBSD 5.4-Stable running on a dual Opteron AMD64 server (Sunfire
v20z) with 2Gig RAM
I'm using RRDtool 1.2.8  from the FreeBSD ports tree.

One interesting note is that originally rrdtool would core dump on a
bus error with the large input to stdin.  I re-built the port using
'-g' as an additional CFLAGS argument to build in debugging symbols,
and now rrdtool no longer dumps core - it just errors as follows:

Note: rrdtool works beautifully with the same command structure and
just fewer graphs to summarize (a piped input of more than 8k will
work just fine)

the error message:
---snip---
# cat rrdinput.txt | rrdtool -
597x254
OK u:0.04 s:0.01 r:0.05
ERROR: unknown function ': '
OK u:0.04 s:0.01 r:0.05
---snip---

what rrdinput.txt has in it (all one line - sorry, it's LONG):
---snip---
graph '/usr/local/jffnms/engine/temp/429674b90045c.dat'
--imgformat=PNG --start=1117070100 --end=1117156500 --base=1000
--lower-limit=1 --title='Traffic Aggregation' --alt-autoscale-max
--color=GRID#CCCCCC --color=MGRID#777777 --height=175 --width=500
--vertical-label='Bits per Second'
DEF:input4=/usr/local/jffnms/rrd/interface-4-0.rrd:data:AVERAGE
DEF:output4=/usr/local/jffnms/rrd/interface-4-1.rrd:data:AVERAGE
DEF:input6=/usr/local/jffnms/rrd/interface-6-0.rrd:data:AVERAGE
DEF:output6=/usr/local/jffnms/rrd/interface-6-1.rrd:data:AVERAGE
DEF:input7=/usr/local/jffnms/rrd/interface-7-0.rrd:data:AVERAGE
DEF:output7=/usr/local/jffnms/rrd/interface-7-1.rrd:data:AVERAGE
DEF:input9=/usr/local/jffnms/rrd/interface-9-0.rrd:data:AVERAGE
DEF:output9=/usr/local/jffnms/rrd/interface-9-1.rrd:data:AVERAGE
DEF:input10=/usr/local/jffnms/rrd/interface-10-0.rrd:data:AVERAGE
DEF:output10=/usr/local/jffnms/rrd/interface-10-1.rrd:data:AVERAGE
DEF:input11=/usr/local/jffnms/rrd/interface-11-0.rrd:!
 data:AVERAGE DEF:output11=/usr/local/jffnms/rrd/interface-11-1.rrd:data:AVERAGE
DEF:input12=/usr/local/jffnms/rrd/interface-12-0.rrd:data:AVERAGE
DEF:output12=/usr/local/jffnms/rrd/interface-12-1.rrd:data:AVERAGE
DEF:input13=/usr/local/jffnms/rrd/interface-13-0.rrd:data:AVERAGE
DEF:output13=/usr/local/jffnms/rrd/interface-13-1.rrd:data:AVERAGE
DEF:input14=/usr/local/jffnms/rrd/interface-14-0.rrd:data:AVERAGE
DEF:output14=/usr/local/jffnms/rrd/interface-14-1.rrd:data:AVERAGE
DEF:input15=/usr/local/jffnms/rrd/interface-15-0.rrd:data:AVERAGE
DEF:output15=/usr/local/jffnms/rrd/interface-15-1.rrd:data:AVERAGE
DEF:input16=/usr/local/jffnms/rrd/interface-16-0.rrd:data:AVERAGE
DEF:output16=/usr/local/jffnms/rrd/interface-16-1.rrd:data:AVERAGE
DEF:input17=/usr/local/jffnms/rrd/interface-17-0.rrd:data:AVERAGE
DEF:output17=/usr/local/jffnms/rrd/interface-17-1.rrd:data:AVERAGE
DEF:input18=/usr/local/jffnms/rrd/interface-18-0.rrd:data:AVERAGE
DEF:output18=/usr/local/jffnms/rrd/interface!
 -18-1.rrd:data:AVERAGE DEF:input19=/usr/local/jffnms/rrd/inter!
 face-19-
0.rrd:data:AVERAGE
DEF:output19=/usr/local/jffnms/rrd/interface-19-1.rrd:data:AVERAGE
DEF:input20=/usr/local/jffnms/rrd/interface-20-0.rrd:data:AVERAGE
DEF:output20=/usr/local/jffnms/rrd/interface-20-1.rrd:data:AVERAGE
DEF:input21=/usr/local/jffnms/rrd/interface-21-0.rrd:data:AVERAGE
DEF:output21=/usr/local/jffnms/rrd/interface-21-1.rrd:data:AVERAGE
DEF:input22=/usr/local/jffnms/rrd/interface-22-0.rrd:data:AVERAGE
DEF:output22=/usr/local/jffnms/rrd/interface-22-1.rrd:data:AVERAGE
DEF:input23=/usr/local/jffnms/rrd/interface-23-0.rrd:data:AVERAGE
DEF:output23=/usr/local/jffnms/rrd/interface-23-1.rrd:data:AVERAGE
DEF:input24=/usr/local/jffnms/rrd/interface-24-0.rrd:data:AVERAGE
DEF:output24=/usr/local/jffnms/rrd/interface-24-1.rrd:data:AVERAGE
DEF:input25=/usr/local/jffnms/rrd/interface-25-0.rrd:data:AVERAGE
DEF:output25=/usr/local/jffnms/rrd/interface-25-1.rrd:data:AVERAGE
DEF:input26=/usr/local/jffnms/rrd/interface-26-0.rrd:data:AVERAGE
DEF:output26=/usr/local/jffnms/rrd/inte!
 rface-26-1.rrd:data:AVERAGE
DEF:input27=/usr/local/jffnms/rrd/interface-27-0.rrd:data:AVERAGE
DEF:output27=/usr/local/jffnms/rrd/interface-27-1.rrd:data:AVERAGE
DEF:input28=/usr/local/jffnms/rrd/interface-28-0.rrd:data:AVERAGE
DEF:output28=/usr/local/jffnms/rrd/interface-28-1.rrd:data:AVERAGE
DEF:input29=/usr/local/jffnms/rrd/interface-29-0.rrd:data:AVERAGE
DEF:output29=/usr/local/jffnms/rrd/interface-29-1.rrd:data:AVERAGE
DEF:input30=/usr/local/jffnms/rrd/interface-30-0.rrd:data:AVERAGE
DEF:output30=/usr/local/jffnms/rrd/interface-30-1.rrd:data:AVERAGE
DEF:input31=/usr/local/jffnms/rrd/interface-31-0.rrd:data:AVERAGE
DEF:output31=/usr/local/jffnms/rrd/interface-31-1.rrd:data:AVERAGE
DEF:input32=/usr/local/jffnms/rrd/interface-32-0.rrd:data:AVERAGE
DEF:output32=/usr/local/jffnms/rrd/interface-32-1.rrd:data:AVERAGE
DEF:input33=/usr/local/jffnms/rrd/interface-33-0.rrd:data:AVERAGE
DEF:output33=/usr/local/jffnms/rrd/interface-33-1.rrd:data:AVERAGE
DEF:input34=/usr/local/jffnms!
 /rrd/interface-34-0.rrd:data:AVERAGE DEF:output34=/usr/local/j!
 ffnms/rr
d/interface-34-1.rrd:data:AVERAGE
DEF:input35=/usr/local/jffnms/rrd/interface-35-0.rrd:data:AVERAGE
DEF:output35=/usr/local/jffnms/rrd/interface-35-1.rrd:data:AVERAGE
DEF:input36=/usr/local/jffnms/rrd/interface-36-0.rrd:data:AVERAGE
DEF:output36=/usr/local/jffnms/rrd/interface-36-1.rrd:data:AVERAGE
DEF:input37=/usr/local/jffnms/rrd/interface-37-0.rrd:data:AVERAGE
DEF:output37=/usr/local/jffnms/rrd/interface-37-1.rrd:data:AVERAGE
DEF:input38=/usr/local/jffnms/rrd/interface-38-0.rrd:data:AVERAGE
DEF:output38=/usr/local/jffnms/rrd/interface-38-1.rrd:data:AVERAGE
DEF:input39=/usr/local/jffnms/rrd/interface-39-0.rrd:data:AVERAGE
DEF:output39=/usr/local/jffnms/rrd/interface-39-1.rrd:data:AVERAGE
DEF:input40=/usr/local/jffnms/rrd/interface-40-0.rrd:data:AVERAGE
DEF:output40=/usr/local/jffnms/rrd/interface-40-1.rrd:data:AVERAGE
DEF:input41=/usr/local/jffnms/rrd/interface-41-0.rrd:data:AVERAGE
DEF:output41=/usr/local/jffnms/rrd/interface-41-1.rrd:data:AVERAGE
DEF:input42=/usr/local/j!
 ffnms/rrd/interface-42-0.rrd:data:AVERAGE
DEF:output42=/usr/local/jffnms/rrd/interface-42-1.rrd:data:AVERAGE
DEF:input43=/usr/local/jffnms/rrd/interface-43-0.rrd:data:AVERAGE
DEF:output43=/usr/local/jffnms/rrd/interface-43-1.rrd:data:AVERAGE
DEF:input45=/usr/local/jffnms/rrd/interface-45-0.rrd:data:AVERAGE
DEF:output45=/usr/local/jffnms/rrd/interface-45-1.rrd:data:AVERAGE
DEF:input46=/usr/local/jffnms/rrd/interface-46-0.rrd:data:AVERAGE
DEF:output46=/usr/local/jffnms/rrd/interface-46-1.rrd:data:AVERAGE
DEF:input47=/usr/local/jffnms/rrd/interface-47-0.rrd:data:AVERAGE
DEF:output47=/usr/local/jffnms/rrd/interface-47-1.rrd:data:AVERAGE
DEF:input49=/usr/local/jffnms/rrd/interface-49-0.rrd:data:AVERAGE
DEF:output49=/usr/local/jffnms/rrd/interface-49-1.rrd:data:AVERAGE
DEF:input50=/usr/local/jffnms/rrd/interface-50-0.rrd:data:AVERAGE
DEF:output50=/usr/local/jffnms/rrd/interface-50-1.rrd:data:AVERAGE
DEF:input51=/usr/local/jffnms/rrd/interface-51-0.rrd:data:AVERAGE
DEF:output51=/u!
 sr/local/jffnms/rrd/interface-51-1.rrd:data:AVERAGE DEF:input5!
 2=/usr/l
ocal/jffnms/rrd/interface-52-0.rrd:data:AVERAGE
DEF:output52=/usr/local/jffnms/rrd/interface-52-1.rrd:data:AVERAGE
DEF:input53=/usr/local/jffnms/rrd/interface-53-0.rrd:data:AVERAGE
DEF:output53=/usr/local/jffnms/rrd/interface-53-1.rrd:data:AVERAGE
DEF:input165=/usr/local/jffnms/rrd/interface-165-0.rrd:data:AVERAGE
DEF:output165=/usr/local/jffnms/rrd/interface-165-1.rrd:data:AVERAGE
DEF:input1184=/usr/local/jffnms/rrd/interface-1184-0.rrd:data:AVERAGE
DEF:output1184=/usr/local/jffnms/rrd/interface-1184-1.rrd:data:AVERAGE
DEF:input1214=/usr/local/jffnms/rrd/interface-1214-0.rrd:data:AVERAGE
DEF:output1214=/usr/local/jffnms/rrd/interface-1214-1.rrd:data:AVERAGE
DEF:input1215=/usr/local/jffnms/rrd/interface-1215-0.rrd:data:AVERAGE
DEF:output1215=/usr/local/jffnms/rrd/interface-1215-1.rrd:data:AVERAGE
DEF:input1216=/usr/local/jffnms/rrd/interface-1216-0.rrd:data:AVERAGE
DEF:output1216=/usr/local/jffnms/rrd/interface-1216-1.rrd:data:AVERAGE
CDEF:input=0,input4,UN,0,input4,IF,+,inp!
 ut6,UN,0,input6,IF,+,input7,UN,0,input7,IF,+,input9,UN,0,input9,IF,+,input10,UN,0,input10,IF,+,input11,UN,0,input11,IF,+,input12,UN,0,input12,IF,+,input13,UN,0,input13,IF,+,input14,UN,0,input14,IF,+,input15,UN,0,input15,IF,+,input16,UN,0,input16,IF,+,input17,UN,0,input17,IF,+,input18,UN,0,input18,IF,+,input19,UN,0,input19,IF,+,input20,UN,0,input20,IF,+,input21,UN,0,input21,IF,+,input22,UN,0,input22,IF,+,input23,UN,0,input23,IF,+,input24,UN,0,input24,IF,+,input25,UN,0,input25,IF,+,input26,UN,0,input26,IF,+,input27,UN,0,input27,IF,+,input28,UN,0,input28,IF,+,input29,UN,0,input29,IF,+,input30,UN,0,input30,IF,+,input31,UN,0,input31,IF,+,input32,UN,0,input32,IF,+,input33,UN,0,input33,IF,+,input34,UN,0,input34,IF,+,input35,UN,0,input35,IF,+,input36,UN,0,input36,IF,+,input37,UN,0,input37,IF,+,input38,UN,0,input38,IF,+,input39,UN,0,input39,IF,+,input40,UN,0,input40,IF,+,input41,UN,0,input41,IF,+,input42,UN,0,input42,IF,+,input43,UN,0,input43,IF,+,input45,UN,0,input45,IF,+,input46,U!
 N,0,input46,IF,+,input47,UN,0,input47,IF,+,input49,UN,0,input4!
 9,IF,+,i
nput50,UN,0,input50,IF,+,input51,UN,0,input51,IF,+,input52,UN,0,input52,IF,+,input53,UN,0,input53,IF,+,input165,UN,0,input165,IF,+,input1184,UN,0,input1184,IF,+,input1214,UN,0,input1214,IF,+,input1215,UN,0,input1215,IF,+,input1216,UN,0,input1216,IF,+,
CDEF:output=0,output4,UN,0,output4,IF,+,output6,UN,0,output6,IF,+,output7,UN,0,output7,IF,+,output9,UN,0,output9,IF,+,output10,UN,0,output10,IF,+,output11,UN,0,output11,IF,+,output12,UN,0,output12,IF,+,output13,UN,0,output13,IF,+,output14,UN,0,output14,IF,+,output15,UN,0,output15,IF,+,output16,UN,0,output16,IF,+,output17,UN,0,output17,IF,+,output18,UN,0,output18,IF,+,output19,UN,0,output19,IF,+,output20,UN,0,output20,IF,+,output21,UN,0,output21,IF,+,output22,UN,0,output22,IF,+,output23,UN,0,output23,IF,+,output24,UN,0,output24,IF,+,output25,UN,0,output25,IF,+,output26,UN,0,output26,IF,+,output27,UN,0,output27,IF,+,output28,UN,0,output28,IF,+,output29,UN,0,output29,IF,+,output30,UN,0,output30,IF,+,output31,UN,0,output31,IF,+,out!
 put32,UN,0,output32,IF,+,output33,UN,0,output33,IF,+,output34,UN,0,output34,IF,+,output35,UN,0,output35,IF,+,output36,UN,0,output36,IF,+,output37,UN,0,output37,IF,+,output38,UN,0,output38,IF,+,output39,UN,0,output39,IF,+,output40,UN,0,output40,IF,+,output41,UN,0,output41,IF,+,output42,UN,0,output42,IF,+,output43,UN,0,output43,IF,+,output45,UN,0,output45,IF,+,output46,UN,0,output46,IF,+,output47,UN,0,output47,IF,+,output49,UN,0,output49,IF,+,output50,UN,0,output50,IF,+,output51,UN,0,output51,IF,+,output52,UN,0,output52,IF,+,output53,UN,0,output53,IF,+,output165,UN,0,output165,IF,+,output1184,UN,0,output1184,IF,+,output1214,UN,0,output1214,IF,+,output1215,UN,0,output1215,IF,+,output1216,UN,0,output1216,IF,+,
CDEF:inputbits=input,UN,0,input,IF,8,*
CDEF:outputbits=output,UN,0,output,IF,8,* HRULE:51000000000#FF0000:' '
COMMENT:'Inbound Bandwidth\: 51000000 kbps' HRULE:51000000000#AA0000:'
' COMMENT:'Outbound Bandwidth\: 51000000 kbps\n' 
AREA:inputbits#00CC00:'Inbound ' GPRINT:i!
 nputbits:MAX:'Max\:%8.2lf %sbps' GPRINT:inputbits:AVERAGE:'Ave!
 rage\:%8
2lf %sbps' GPRINT:inputbits:LAST:'Last\:%8.2lf %sbps\n'
LINE2:outputbits#0000FF:Outbound GPRINT:outputbits:MAX:'Max\:%8.2lf
%sbps' GPRINT:outputbits:AVERAGE:'Average\:%8.2lf %sbps'
GPRINT:outputbits:LAST:'Last\:%8.2lf %sbps\n'
---snip---

I suspect the issue is somewhere in rrd_open.c (I think that was the
file name) where stdin is chunked in to blocks of 8000-and-change
bytes and read in, but I can't be sure as rrdtool is no longer dumping
core with debugging symbols and I can't run gdb against a command
reading input from stdin =)

I'd appreciate any debugging tips or pointers.

Thanks,
     ~Aaron

--
Unsubscribe mailto:rrd-users-request at list.ee.ethz.ch?subject=unsubscribe
Help        mailto:rrd-users-request at list.ee.ethz.ch?subject=help
Archive     http://lists.ee.ethz.ch/rrd-users
WebAdmin    http://lists.ee.ethz.ch/lsg2.cgi