[rrd-developers] rrdcached contention when flushing

[kevin brintnall]

On Tue, Nov 04, 2008 at 10:22:52PM +0000, Daniel Pocock wrote:
> > That sill should not cause your process to bloat to several gigabytes..
> > The size of the tree should be on the order of O(node_count * PATH_MAX).
> >   
> > What does "STATS" from your daemon look like?
>
> After seeing the valgrind output - which only indicates a static amount
> of un-freed memory relating to keys, as you have described - I'm
> thinking that the problem may not be a leak, rather, it may be some kind
> of backlog of updates.  I've got a few more tests planned to investigate
> this, for example, slowing down the IO to see if memory usage grows more
> quickly.

You can estimate the memory utilization like this:

 num_rrds * (file_name_length + (w_timer / rrd_step) * update_str_len)

If you want to see what's outstanding for a few files, you can use
"PENDING /path/file"  and see what the daemon has.

> Can you think of anything extra that could be added to the STATS output 
> to indicate whether there is a backlog, how long it takes to come back 
> to each node, etc?
>
> Is it possible that some files would never get flushed, if the flush 
> command was being invoked repeatedly for a subset of the files?

It's possible..  Periodically enqueued files are placed on the TAIL,
whereas "FLUSH"ed files are placed on the HEAD.  It's possible to starve
the normal update (both -w and -f timers) with a lot of FLUSH.

If that's happening, you will see your "QueueLength" stat stay >0.

-- 
 kevin brintnall =~ /kbrint at rufus.net/