[rrd-users] Incremental backup rrd file

[Tobias Oetiker]

Hi Ccddtt,

Yesterday ccddtt wrote:

> Hi,
>
> Thanks!
> Very sorry, I did not carefully read the RSYNC document.I have just tested, RSYNC is really only to send the updated file block.
> very grateful to you for help, and this solves the problem I wrote the code.(this is a happy thing)
>
> By the way, RSYNC command parameters I use the rsync -azh ,
> the -z is
> -z, --compress              compress file data during the transfer
>
> I think use this good.
>
> and I not use ssh over rsync. ssh encrypted data packet, it will increase my traffic.
>
> Thanks!

also note, that rsync can do 'in-place' updates, then it will only
modify the changed block and not copy the entire rrd file at the
destination which might be helful.

cheers
tobi

>
>
>
>
> 2012-07-18
>
>
>
>
>
> ???? Ryan Kubica
> ????? 2012-07-18 00:05:47
> ???? Simon Hobson; rrd-users at lists.oetiker.ch
> ???
> ??? Re: [rrd-users] Incremental backup rrd file
>
>
>
> I'll second all of this about rsync: it's very efficient and 'safe' for rrd data copies.
>
>
> I don't have backups per-se, I run active mirrored rrd servers with millions of rrd datafiles per server and if one crashes where I need to rebuild one or install a new one for hardware upgrade like I'm doing today, then I use rsync to get a copy from another mirror ... actively.  The replacement-mirror writes behind in the rrd update queue so it's updating older intervals than the rest of the cluster and then I copy from another mirror.  I'm currently copying 1TB (one terabyte) and it works beautifully.
>
>
> rsync would take a long time to do backups nightly of that many files (which is why it's not done); but on a few thousand'ish it can(should!) be used.
>
>
> If you use rsync over ssh, at least do something like this: rsync -ave 'ssh -c blowfish' src dst
>
>
> I've yet to bother with rsync daemon with no ssh, though that'd be more efficient as well.
>
>
> -Ryan
>
>
>
>
> From: Simon Hobson <linux at thehobsons.co.uk>
> To: rrd-users at lists.oetiker.ch
> Sent: Tuesday, July 17, 2012 4:47 AM
> Subject: Re: [rrd-users] Incremental backup rrd file
>
>
> Darren Murphy wrote:
> >Just to add a little to this, the --stats & --human-readable options
> >provide useful insight as to the efficiency of rsync
> <snip>
> >So 3121 files totaling 4.3GB in size, and at least 90% of those files
> >would change between successive sync runs, yet only a very small
> >amount of data needs to be transferred.
>
> That tallies with my experience. Obviously it varies considerably
> with the type of data, but I've yet to find something where it
> doesn't show a reduction in data transferred.
> In general, RRD files should 'compress' quite well (unless you use
> very small consolidations).
>
> >I'd also add that in my experience rsync is incredibly robust and reliable.
> >I've been running an hourly rsync from my main MRTG server to 3
> >separate "slaves" for almost 2 years now, and never once had a problem
> >with data integrity.
>
> I'll second that. And of course, even if the process dies part way
> through, you can just run it again and it will catch up.
>
>

-- 
Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland
http://it.oetiker.ch tobi at oetiker.ch ++41 62 775 9902 / sb: -9900