[rrd-developers] rrdcached contention when flushing
kbrint at rufus.net
Tue Nov 4 23:34:45 CET 2008
On Tue, Nov 04, 2008 at 10:22:52PM +0000, Daniel Pocock wrote:
> > That sill should not cause your process to bloat to several gigabytes..
> > The size of the tree should be on the order of O(node_count * PATH_MAX).
> > What does "STATS" from your daemon look like?
> After seeing the valgrind output - which only indicates a static amount
> of un-freed memory relating to keys, as you have described - I'm
> thinking that the problem may not be a leak, rather, it may be some kind
> of backlog of updates. I've got a few more tests planned to investigate
> this, for example, slowing down the IO to see if memory usage grows more
You can estimate the memory utilization like this:
num_rrds * (file_name_length + (w_timer / rrd_step) * update_str_len)
If you want to see what's outstanding for a few files, you can use
"PENDING /path/file" and see what the daemon has.
> Can you think of anything extra that could be added to the STATS output
> to indicate whether there is a backlog, how long it takes to come back
> to each node, etc?
> Is it possible that some files would never get flushed, if the flush
> command was being invoked repeatedly for a subset of the files?
It's possible.. Periodically enqueued files are placed on the TAIL,
whereas "FLUSH"ed files are placed on the HEAD. It's possible to starve
the normal update (both -w and -f timers) with a lot of FLUSH.
If that's happening, you will see your "QueueLength" stat stay >0.
kevin brintnall =~ /kbrint at rufus.net/
More information about the rrd-developers