[rrd-users] Locking *read* (copy/backup) of RRD files?

[Steve Shipway]

I had a bit of a hack with the rrdcached code yesterday; I added in a PAUSE and RESUME command which temporarily stops and restarts the write threads while still accepting (and queueing) the pending updates, so that nothing is lost during the backups.

Its still rather rough and (1) it pauses EVERYTHING so you have to watch how big the in-memory queue gets, and (2) during my test, there were 10,000+ updates queued over a 20min pause, but for some reason 4 writes were still done.  I haven't yet found out why.  So, its not stable yet, though it's definitely got possibilities.  I can send you a copy of the modified code if you want though I'd certinaly not advise putting it into a production environment yet!

Steve

Steve Shipway
University of Auckland ITS
UNIX Systems Design Lead
s.shipway at auckland.ac.nz<mailto:s.shipway at auckland.ac.nz>
Ph: +64 9 373 7599 ext 86487

________________________________
From: Jochen Bern [Jochen.Bern at LINworks.de]
Sent: Wednesday, 27 October 2010 9:08 p.m.
To: rrd-users at lists.oetiker.ch
Cc: Steve Shipway
Subject: Re: [rrd-users] Locking *read* (copy/backup) of RRD files?

On 10/27/2010 02:32 AM, Steve Shipway wrote:
> however I'd use a filesystem snapshot so the time during which the
> rrdcached is inoperative would be measured in seconds

That is, of course, the authoritative solution to the problem of doing
full backups - and would require a full reinstall in my case because a
need for snapshot capability wasn't anticipated at OS install time. :-}
(Someone else did it, resulting in ext3 on LVM with no unused physical
extents ...)

However, it's not too difficult to imagine scenarios where one would
*still* want consistent duplication on a single-RRD-file level; e.g.,
thanks to n2rrd in full auto mode, I get to go through the cycle of
rrddump;vi;mv;rrdrestore every now and then to remove bogus DSs ...

>> As a matter of fact, I'm a bit surprised to see that rrdresize seems to
>> be the *only* part of rrdtool which sets a(ny) lock while *reading* an
>> RRD file.
> I'd need to check, but possibly since the RRD file size is constant
> (except when running rrdresize) it means that since reads and writes
> are atomic, the reads do not need to be locked as you will always get
> consistent data?

In 1.4.4 src/rrd_open.c::rrd_open(), I see eight __rrd_read() ; these
read out of an mmap() if available, and do an actual read() else. My
mmap(2) manpage (OpenSuSE 11.2) warns me that it "is unspecified whether
changes made to the file after the mmap() call are visible in the mapped
region", not to even dream about any kind of atomicity ...

Kind regards,
                                                                J. Bern
--
Jochen Bern, Systemingenieur --- LINworks GmbH <http://www.LINworks.de/<http://www.linworks.de/>>
Postfach 100121, 64201 Darmstadt | Robert-Koch-Str. 9, 64331 Weiterstadt
PGP (1024D/4096g) FP = D18B 41B1 16C0 11BA 7F8C DCF7 E1D5 FAF4 444E 1C27
Tel. +49 6151 9067-231, Zentr. -0, Fax -299 - Amtsg. Darmstadt HRB 85202
Unternehmenssitz Weiterstadt, Geschäftsführer Metin Dogan, Oliver Michel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.oetiker.ch/pipermail/rrd-users/attachments/20101027/dc7a119d/attachment.htm