[rrd-developers] Locking concept for RRD files, concurrent write/read operations

[Alex van den Bogaerdt]

On Mon, Apr 23, 2007 at 04:08:54PM +0200, Roger J. Meier wrote:
> Hi all,
> 
> There is a LockRRD() function in rrd_update.c which is called only from
> rrd_update() and rrd_resize() functions.  The LockRRD() allocates an exclusive
> write lock (F_WRLCK) on the whole RRD file and fails if it can't lock the
> file.
> 
> Apparently, the concept is to lock the file for each write access, but not for
> reading (no shared read locks).
> 
> In our environment it seems that -- on a heavy loaded system -- the reading
> of RRD data fails from time to time (Linux).
> 
> Now the "usual suspect":
> 
> What if some process writes an RRD file and at the same time another process
> reads it to fetch data or generate a graph? The writing of an RRD file is not
> atomic, and the read operations of the reading process may get a mixture of
> old and new data.  This is possible since the LockRRD() only prevents
> concurrent writings, but allows one process writing a file which may be read
> at the same time by other processes.  There is no read lock acquired.

It has been a while since I looked at the code, but I'll attempt to provide
some input for you anyway.  Take the following with a grain of salt.

Under normal circumstances (rrdtool graph, rrdtool fetch), data will be
read from RRAs.  Conceptually these are wheel-like structures, with a
pointer pointing to the newest entry.  One neighbour of this newest entry
is the next-to-newest one; the other neighbour is the oldest entry.

When "rrdtool update" writes data, it will overwrite the oldest entry,
and then advance the pointer.

"rrdtool graph" will look at the pointer position, subtract some number
of intervals, and read the data inbetween these two positions (inclusive).

If a problem occurs, it will be this one:

a) rrdtool graph looks at the pointer position and computes which entries
   to read from the RRA
b) rrdtool update overwrites the oldest entry
c) rrdtool graph reads the RRA data
d) rrdtool update adjusts the pointer position

This shouldn't result in the kind of problem you're describing.  The worst
what will happen is that you use the oldest entry, which is no longer valid
(it has become the newest entry, but rrdtool graph doesn't know this).


I currently have no explanation for the problem you report. Also I don't
think I can fully picture this problem in my mind.  "Reading fails", but
what exactly does happen?

cheers
-- 
Alex van den Bogaerdt
http://www.vandenbogaerdt.nl/rrdtool/