[mrtg] Performance Problem

[Chuck.Coker at rcc.edu]

Hi Folks,

We are having major performance problems with our MRTG box. We are
currently monitoring 507 routers and switches with 9,621 interfaces.
I probably have another 200-300 routers and switches to add. That's not
too many is it?

When polling these devices at five minute intervals, I run out of swap
space and receive this error message:
    swap_pager_getswapspace: failed

So far, I've increased the polling interval from five minutes to ten
minutes and it has temporarily solved the swap space problems. We get
the occasional gray bar (as in no response) in our graphs, but overall
things are working reasonably well.

When I try to add any more devices, I start getting the swap space
errors again. I suspect that I've reached the limits of what our little
MRTG box can do.

It appears that we have plenty of unused memory, so the bottleneck
should be with something else, for example, running out of buffers. On
the FreeBSD web site, they suggest increasing the kern.ipc.nmbclusters
to somewhere between 1024 to 4096. Right now we have it set to 8960.
Could we have it set too high?

I'm wondering if there is anything that we can tweak to get better
performance. The box only runs MRTG and Apache, so anything I need to do
to make MRTG work better at the expense of something else is okay.

I've included some of the info from netstat, sysctl, and dmesg. We also
have kern.ipc.numopensockets set to 10 and kern.ipc.maxsockets set to
4008.

Thanks,
Chuck

PS: Does anyone know of a place where I can find descriptions what all
the sysctl variables are used for? I mean more of a description than the
output from:
    sysctl -d -a
I'm looking for enough information to make educated guesses about what
can be tweaked and by how much or how little.

======================================================================

$ netstat -m
mbuf usage:
	GEN cache:	0/0 (in use/in pool)
	CPU #0 cache:	129/192 (in use/in pool)
	Total:		129/192 (in use/in pool)
	Mbuf cache high watermark: 512
	Maximum possible: 17920
	Allocated mbuf types:
	  129 mbufs allocated to data
	1% of mbuf map consumed
mbuf cluster usage:
	GEN cache:	0/0 (in use/in pool)
	CPU #0 cache:	128/152 (in use/in pool)
	Total:		128/152 (in use/in pool)
	Cluster cache high watermark: 128
	Maximum possible: 8960
	1% of cluster map consumed
352 KBytes of wired memory reserved (81% in use)
0 requests for memory denied
0 requests for memory delayed
0 calls to protocol drain routines

======================================================================

$ sysctl -A | grep mbuf
kern.ipc.nmbufs: 17920
kern.ipc.mbuf_wait: 64
kern.ipc.mbuf_hiwm: 512
kern.ipc.mbuf_lowm: 128
      mbufmgr    32    79K     79K      459
16,32,64,256,4096,8192,65536

======================================================================

$ dmesg
...
FreeBSD 5.1-RELEASE #0: Thu Jun  5 02:55:42 GMT 2003
    root at wv1u.btc.adaptec.com:/usr/obj/usr/src/sys/GENERIC
Preloaded elf kernel "/boot/kernel/kernel" at 0xc06d4000.
Preloaded elf module "/boot/kernel/acpi.ko" at 0xc06d421c.
Timecounter "i8254"  frequency 1193182 Hz
Timecounter "TSC"  frequency 730949743 Hz
CPU: Intel Pentium III (730.95-MHz 686-class CPU)
    Origin = "GenuineIntel"  Id = 0x683  Stepping = 3
 
Features=0x383fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,
MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE>
real memory  = 268304384 (255 MB)
avail memory = 253231104 (241 MB)
...

----------------------------------------------------------------
Chuck Coker                             chuck.coker (et) rcc.edu
       Riverside Community College, Information Services
4800 Magnolia Avenue, Riverside, California 92506, United States
+1 951 222 8759 (desk)                    +1 951 830 8068 (cell)
----------------------------------------------------------------

--
Unsubscribe mailto:mrtg-request at list.ee.ethz.ch?subject=unsubscribe
Archive     http://www.ee.ethz.ch/~slist/mrtg
FAQ         http://faq.mrtg.org    Homepage     http://www.mrtg.org
WebAdmin    http://www.ee.ethz.ch/~slist/lsg2.cgi