[mrtg] Graphing lots of ports
mr.flarp at gmail.com
Mon Dec 6 22:32:09 MET 2004
I have a pair of Linux machines graphing a lot of switch ports (~35000
ports across 1500 devices each) on 5-minute intervals. Disk I/O seems
to be my biggest problem. The CPU load tends to average about 60-70%
busy. About once every 10-15 minutes, though, the kjournald and
kupdated processes launch and take the machine down with it for about
5 minutes at a time. During this, top shows 'iowait' to be at 100% for
both CPUs, and almost nothing will run on the machine. On a good run,
it takes about 3 minutes to finish. When it lags, sometimes it won't
recover for 10+ minutes.
Each machine has a single MRTG config file that the collector is run
against. The config looks something like this:
----- Begin config ----
# port id: 11536 - switch id: 922 - ifindex: 10
Target[922.11536]: 10:public at 172.16.99.133:::1::2
SetEnv[922.11536]: MRTG_INT_IP="" MRTG_INT_DESCR="2/2"
Title[922.11536]: Port 2/2 -- c2948-4 (172.16.99.133)
# port id: 11537 - switch id: 922 - ifindex: 11
Target[922.11537]: 11:public at 172.16.99.133:::1::2
SetEnv[922.11537]: MRTG_INT_IP="" MRTG_INT_DESCR="2/3"
Title[922.11537]: Port 2/3 -- c2948-4 (172.16.99.133)
... (many more similar entries) ...
----- End config -----
Hardware: Dell PowerEdge 2650 - Dual Xeon @ 3.06GHz, 2GB RAM, Hardware
RAID 5 (PERC-3, aacraid driver), 5x 10000RPM U320 SCSI drives
Software: Redhat AS3, ext3 filesystem, MRTG 2.10.15, RRDTool 1.0.49
So... my question is this. Have any of you tried running similar
loads? And have you experienced similar problems? And do you have any
ideas how to address them? Any other optimizations that might help?
I found that by sorting the individual RRD files into some sort of
directory structure (rather than 35000 files in a single directory)
helped a lot. Sorting by switch ID was just an arbitrary choice. Also,
disabling the set atime on the filesystem (chattr +A) made a little
Unsubscribe mailto:mrtg-request at list.ee.ethz.ch?subject=unsubscribe
FAQ http://faq.mrtg.org Homepage http://www.mrtg.org
More information about the mrtg