[mrtg] Re: bad performance: MRTG 2.9.4 / RRDs / rrdtool 1.0.28

[Rainer Bawidamann]

In article <20010109001401.A3354 at nextra.com>,
	bn at nextra.com (Bjorn Nordbo) writes:
> 
> I have a Sun box with 4 450MHz Ultra IIi's, 2GB RAM which I use for
> collection information through SNMP. The RRD-archives rests on a
> NetApp for performance reasons. But still I get into trouble when
> the config files reaches 500 targets. The collect operation is done
> in about 30 seconds with 8 forks, but the RRDs::update()'s takes a
> very long time to complete, about one second each (I have confirmed
> that MRTG does one RRDs::update() per target).

NetApp - is this a NFS server? NFS is very slow compared to a local
disc. I know of installations with 4400+ targets where RRDs::update is
not a problem.

>       527 188.4178 2.980000   410:        RRDs::update("$rrd",
>      2640 50.81802 0.090000  1581:      my $line = <$h>; # must be a simple
>       527 21.86756 0.550000   417:        my $lasttime = RRDs::last($rrd);
>       527 15.09493 0.300000   402:            RRDs::tune(@args);
>       527 2.734013 0.700000   421:        my @fetch =

Here you will see 4 calls to RRDs: tune (called earlier than in this
listing), update, last, fetch (fetch not listed). The last and fetch
calls could be optimized away if you don't use threshold checking, the
tune call could be skipped with two 'stat' calls before most of the
time. But as you're the first that complains about rrdtool performance
(did you ever use rateup?) I think the problem is NFS.

> Here is the part where RRDs::update() is involved (all of it):

It's more than that:

1. RRDs::tune

> open("/local/net/rrd/misc/ti01a01fastethernet0.rrd", O_RDWR) = 6
> fstat64(6, 0xFFBEF458)                          = 0
> ioctl(6, TCGETA, 0xFFBEF3E4)                    Err#25 ENOTTY
> read(6, " R R D\0 0 0 0 1\0\0\0\0".., 8192)     = 8192
> lseek(6, 0, SEEK_SET)                           = 0
> write(6, " R R D\0 0 0 0 1\0\0\0\0".., 360)     = 360
> close(6)                                        = 0
> time()                                          = 978994523

2. RRDs::update (needs to change every RRA)

> open("/local/net/rrd/misc/ti01a01fastethernet0.rrd", O_RDWR) = 6
> fstat64(6, 0xFFBEF460)                          = 0
> ioctl(6, TCGETA, 0xFFBEF3EC)                    Err#25 ENOTTY
> read(6, " R R D\0 0 0 0 1\0\0\0\0".., 8192)     = 8192
> llseek(6, 0, SEEK_CUR)                          = 8192
> lseek(6, 0xFFFFEC00, SEEK_CUR)                  = 3072
> fcntl(6, F_SETLK, 0xFFBEF5F0)                   = 0
> llseek(6, 0, SEEK_CUR)                          = 3072
> lseek(6, 3104, SEEK_SET)                        = 3104
> write(6, "7FFFFFFFFFFFFFFF7FFFFFFF".., 208)     = 208
> lseek(6, 288784, SEEK_SET)                      = 288784
> write(6, "7FFFFFFFFFFFFFFF7FFFFFFF".., 32)      = 32
> lseek(6, 299984, SEEK_SET)                      = 299984
> write(6, "7FFFFFFFFFFFFFFF7FFFFFFF".., 16)      = 16
> lseek(6, 325152, SEEK_SET)                      = 325152
> write(6, "7FFFFFFFFFFFFFFF7FFFFFFF".., 208)     = 208
> lseek(6, 334736, SEEK_SET)                      = 334736
> write(6, "7FFFFFFFFFFFFFFF7FFFFFFF".., 32)      = 32
> lseek(6, 345936, SEEK_SET)                      = 345936
> write(6, "7FFFFFFFFFFFFFFF7FFFFFFF".., 16)      = 16
> lseek(6, 371104, SEEK_SET)                      = 371104
> write(6, "7FFFFFFFFFFFFFFF7FFFFFFF".., 48)      = 48
> lseek(6, 371072, SEEK_SET)                      = 371072
> write(6, "7FFFFFFFFFFFFFFF7FFFFFFF".., 80)      = 80
> lseek(6, 371072, SEEK_SET)                      = 371072
> write(6, "7FFFFFFFFFFFFFFF7FFFFFFF".., 80)      = 80
> lseek(6, 1368, SEEK_SET)                        = 1368
> write(6, " : Z E [ + 9 1 7 0 7 8 6".., 1704)    = 1704
> close(6)                                        = 0

3. RRDs::last (get the time of the last complete entry)

> open("/local/net/rrd/misc/ti01a01fastethernet0.rrd", O_RDONLY) = 6
> fstat64(6, 0xFFBEF4C0)                          = 0
> ioctl(6, TCGETA, 0xFFBEF44C)                    Err#25 ENOTTY
> read(6, " R R D\0 0 0 0 1\0\0\0\0".., 8192)     = 8192
> llseek(6, 0xFFFFFFFFFFFFEC00, SEEK_CUR)         = 3072
> close(6)                                        = 0
> time()                                          = 978994524
> time()                                          = 978994524
> time()                                          = 978994524
> time()                                          = 978994524

4. RRDs::fetch (fetch the last entry for threshold checking)

> open("/local/net/rrd/misc/ti01a01fastethernet0.rrd", O_RDONLY) = 6
> fstat64(6, 0xFFBEF3B0)                          = 0
> ioctl(6, TCGETA, 0xFFBEF33C)                    Err#25 ENOTTY
> read(6, " R R D\0 0 0 0 1\0\0\0\0".., 8192)     = 8192
> llseek(6, 0, SEEK_CUR)                          = 8192
> lseek(6, 288992, SEEK_SET)                      = 288992
> lseek(6, 3296, SEEK_SET)                        = 3296
> read(6, "7FFFFFFFFFFFFFFF7FFFFFFF".., 8192)     = 8192
> llseek(6, 0xFFFFFFFFFFFFE010, SEEK_CUR)         = 3312
> close(6)                                        = 0

> This seems awfully ineffective. Four opens, 11 writes and a whole
> bunch of seeks. Does anyone know what rrdtool is doing here that
> requires so many operations just for a simple update?

The seeks should improve performance on local discs as fewer reads are
necessary. Maybe more reads/fewer seeks would be better for NFS.

The multiple opens/reads should be cached by the file system in memory.
It's possible that NFS doesn't handle this well (especially the writes
inbetween).

So my advice is to use a local disc instead of NFS for the archives. It
shouldn't be a problem to access the archvies via NFS on the web server
(were the images are generated) as 14all just reads the rrd files.

Rainer

--
Unsubscribe mailto:mrtg-request at list.ee.ethz.ch?subject=unsubscribe
Archive     http://www.ee.ethz.ch/~slist/mrtg
FAQ         http://faq.mrtg.org    Homepage     http://www.mrtg.org
WebAdmin    http://www.ee.ethz.ch/~slist/lsg2.cgi