"Raimund Berger" <raimund.berger at gmail.com> writes:
> open("/var/lib/munin/quasi.internal/router.quasi.internal-if_eth0-up-c.rrd", O_RDONLY) = 4
> fstat64(4, {st_mode=S_IFREG|0644, st_size=50604, ...}) = 0
> fadvise64(4, 0, 0, POSIX_FADV_RANDOM) = 0
> mmap2(NULL, 50604, PROT_READ, MAP_PRIVATE|MAP_NORESERVE, 4, 0) = 0xb78c9000
> madvise(0xb78c9000, 50604, 0x1 /* MADV_??? */) = 0
> madvise(0xb78c9000, 112, MADV_SEQUENTIAL|0x1) = 0
> madvise(0xb78c9000, 120, MADV_SEQUENTIAL|0x1) = 0
> madvise(0xb78c9000, 1296, MADV_SEQUENTIAL|0x1) = 0
> madvise(0xb78c9000, 4, MADV_SEQUENTIAL|0x1) = 0
> msync(0xb78c9000, 50604, MS_ASYNC) = 0
> munmap(0xb78c9000, 50604) = 0
> close(4) = 0
> time(NULL) = 1213703114
> mmap2(NULL, 1073152, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb77d0000
> brk(0x80b1000) = 0x80b1000
> brk(0x80d9000) = 0x80d9000
> mmap2(NULL, 266240, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb778f000
> brk(0x80b8000) = 0x80b8000
> mmap2(NULL, 528384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb770e000
> munmap(0xb778f000, 266240) = 0
> mmap2(NULL, 1052672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb760d000
> munmap(0xb770e000, 528384) = 0
> mmap2(NULL, 2101248, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb740c000
> munmap(0xb760d000, 1052672) = 0
> mmap2(NULL, 4198400, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb700b000
> munmap(0xb740c000, 2101248) = 0
> mmap2(NULL, 8392704, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb680a000
> munmap(0xb700b000, 4198400) = 0
> mmap2(NULL, 16781312, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb5809000
> munmap(0xb680a000, 8392704) = 0
> mmap2(NULL, 33558528, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb3808000
> munmap(0xb5809000, 16781312) = 0
> mmap2(NULL, 67112960, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xaf807000
> munmap(0xb3808000, 33558528) = 0
> mmap2(NULL, 134221824, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xa7806000
> munmap(0xaf807000, 67112960) = 0
> mmap2(NULL, 268439552, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x97805000
> munmap(0xa7806000, 134221824) = 0
> mmap2(NULL, 536875008, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x77804000
>
> that is, mmap going and allocating memory to powers of 2, doubling
> each time.
>
> Finally, an interesting observation. I tried that same call on older
> files, and it worked allright. The same applies if I just change the
> start time in the above call and on the same file to 9am-1day, while
> just 9am gives the said memory problem. Thought this might be of
> interest.
>
> Let me know if you'd like me to provide more input.
>
> R.
OK, some further info.
(1) "powers of 2" was of course wrong, didn't look too closely
(2) recent runs were ok, and I found the problem pertain to some
specific time interval, likely related to some "problematic" data
stored during that interval in the rrd, whatever that might mean
(3) rrdtool fetch runs ok on the same interval were graph fails
(4) I initially thought it was rrdtools' mmap which gave the trouble,
but from a quick look at the source there seems to be only a single
mmap call in rrd_open, and the strace says that goes alright (file
open, mmap, file close). So right now, I'm not sure what might go
wrong here.
I also did an ltrace but couldn't immediately see if any/what other
library might be involved in this error. Wonder if it's related to the
new (?) cairo/pango stuff.
Sorry for not being able to more accurately pinpoint this. As said,
the system runs alright in all other respects, as well as the default
rrdtool install does, and I guess I'll leave it at that for now.
Thanks, R.