[rrd-users] bad performance: MRTG 2.9.4 / RRDs / rrdtool 1.0.28

[Bjorn Nordbo]

[originally posted on the MRTG list, but it probably belongs here]

I have a Sun box with 4 450MHz Ultra IIi's, 2GB RAM which I use for
collection information through SNMP. The RRD-archives rests on a
NetApp for performance reasons. But still I get into trouble when
the config files reaches 500 targets. The collect operation is done
in about 30 seconds with 8 forks, but the RRDs::update()'s takes a
very long time to complete, about one second each (I have confirmed
that MRTG does one RRDs::update() per target).

This is the head of the sorted output from the profiler:

---8<---
      527 188.4178 2.980000   410:        RRDs::update("$rrd",
     2640 50.81802 0.090000  1581:      my $line = <$h>; # must be a simple
      527 21.86756 0.550000   417:        my $lasttime = RRDs::last($rrd);
      527 15.09493 0.300000   402:            RRDs::tune(@args);
      527 2.734013 0.700000   421:        my @fetch =
---8<---

No problem to see where the problem is, so I did a truss of a MRTG-
run with only *one* target. Here is the part where RRDs::update() is
involved (all of it):

---8<---
open("/local/net/rrd/misc/ti01a01fastethernet0.rrd", O_RDWR) = 6
fstat64(6, 0xFFBEF458)                          = 0
ioctl(6, TCGETA, 0xFFBEF3E4)                    Err#25 ENOTTY
read(6, " R R D\0 0 0 0 1\0\0\0\0".., 8192)     = 8192
lseek(6, 0, SEEK_SET)                           = 0
write(6, " R R D\0 0 0 0 1\0\0\0\0".., 360)     = 360
close(6)                                        = 0
time()                                          = 978994523
open("/local/net/rrd/misc/ti01a01fastethernet0.rrd", O_RDWR) = 6
fstat64(6, 0xFFBEF460)                          = 0
ioctl(6, TCGETA, 0xFFBEF3EC)                    Err#25 ENOTTY
read(6, " R R D\0 0 0 0 1\0\0\0\0".., 8192)     = 8192
llseek(6, 0, SEEK_CUR)                          = 8192
lseek(6, 0xFFFFEC00, SEEK_CUR)                  = 3072
fcntl(6, F_SETLK, 0xFFBEF5F0)                   = 0
llseek(6, 0, SEEK_CUR)                          = 3072
lseek(6, 3104, SEEK_SET)                        = 3104
write(6, "7FFFFFFFFFFFFFFF7FFFFFFF".., 208)     = 208
lseek(6, 288784, SEEK_SET)                      = 288784
write(6, "7FFFFFFFFFFFFFFF7FFFFFFF".., 32)      = 32
lseek(6, 299984, SEEK_SET)                      = 299984
write(6, "7FFFFFFFFFFFFFFF7FFFFFFF".., 16)      = 16
lseek(6, 325152, SEEK_SET)                      = 325152
write(6, "7FFFFFFFFFFFFFFF7FFFFFFF".., 208)     = 208
lseek(6, 334736, SEEK_SET)                      = 334736
write(6, "7FFFFFFFFFFFFFFF7FFFFFFF".., 32)      = 32
lseek(6, 345936, SEEK_SET)                      = 345936
write(6, "7FFFFFFFFFFFFFFF7FFFFFFF".., 16)      = 16
lseek(6, 371104, SEEK_SET)                      = 371104
write(6, "7FFFFFFFFFFFFFFF7FFFFFFF".., 48)      = 48
lseek(6, 371072, SEEK_SET)                      = 371072
write(6, "7FFFFFFFFFFFFFFF7FFFFFFF".., 80)      = 80
lseek(6, 371072, SEEK_SET)                      = 371072
write(6, "7FFFFFFFFFFFFFFF7FFFFFFF".., 80)      = 80
lseek(6, 1368, SEEK_SET)                        = 1368
write(6, " : Z E [ + 9 1 7 0 7 8 6".., 1704)    = 1704
close(6)                                        = 0
open("/local/net/rrd/misc/ti01a01fastethernet0.rrd", O_RDONLY) = 6
fstat64(6, 0xFFBEF4C0)                          = 0
ioctl(6, TCGETA, 0xFFBEF44C)                    Err#25 ENOTTY
read(6, " R R D\0 0 0 0 1\0\0\0\0".., 8192)     = 8192
llseek(6, 0xFFFFFFFFFFFFEC00, SEEK_CUR)         = 3072
close(6)                                        = 0
time()                                          = 978994524
time()                                          = 978994524
time()                                          = 978994524
time()                                          = 978994524
open("/local/net/rrd/misc/ti01a01fastethernet0.rrd", O_RDONLY) = 6
fstat64(6, 0xFFBEF3B0)                          = 0
ioctl(6, TCGETA, 0xFFBEF33C)                    Err#25 ENOTTY
read(6, " R R D\0 0 0 0 1\0\0\0\0".., 8192)     = 8192
llseek(6, 0, SEEK_CUR)                          = 8192
lseek(6, 288992, SEEK_SET)                      = 288992
lseek(6, 3296, SEEK_SET)                        = 3296
read(6, "7FFFFFFFFFFFFFFF7FFFFFFF".., 8192)     = 8192
llseek(6, 0xFFFFFFFFFFFFE010, SEEK_CUR)         = 3312
close(6)                                        = 0
---8<---

This seems awfully ineffective. Four opens, 11 writes and a whole
bunch of seeks. Does anyone know what rrdtool is doing here that
requires so many operations just for a simple update?

Sorry if this message is a little messy, but I've been at work
for 15 hours now..

Any help greatly appreciated!

-- 
Bjørn Nordbø  -  IP Development  -  Nextra Norway

--
Unsubscribe mailto:rrd-users-request at list.ee.ethz.ch?subject=unsubscribe
Help        mailto:rrd-users-request at list.ee.ethz.ch?subject=help
Archive     http://www.ee.ethz.ch/~slist/rrd-users
WebAdmin    http://www.ee.ethz.ch/~slist/lsg2.cgi