[rrd-users] Incremental backup rrd file

[Darren Murphy]

Just to add a little to this, the --stats & --human-readable options
provide useful insight as to the efficiency of rsync

I use rsync on my own (mrtg) rrd files with those options enabled, and
the output of a typical run looks like so:

Jul 17 16:50:04 babylon-db-1 rsync-mrtg[32244]: Started
Jul 17 16:54:53 babylon-db-1 rsync-mrtg[32246]:
Jul 17 16:54:53 babylon-db-1 rsync-mrtg[32246]: Number of files: 3121
Jul 17 16:54:53 babylon-db-1 rsync-mrtg[32246]: Number of files
transferred: 2552
Jul 17 16:54:53 babylon-db-1 rsync-mrtg[32246]: Total file size: 4.30G bytes
Jul 17 16:54:53 babylon-db-1 rsync-mrtg[32246]: Total transferred file
size: 3.51G bytes
Jul 17 16:54:53 babylon-db-1 rsync-mrtg[32246]: Literal data: 9.31M bytes
Jul 17 16:54:53 babylon-db-1 rsync-mrtg[32246]: Matched data: 3.51G bytes
Jul 17 16:54:53 babylon-db-1 rsync-mrtg[32246]: File list size: 73318
Jul 17 16:54:53 babylon-db-1 rsync-mrtg[32246]: File list generation
time: 0.112 seconds
Jul 17 16:54:53 babylon-db-1 rsync-mrtg[32246]: File list transfer
time: 0.000 seconds
Jul 17 16:54:53 babylon-db-1 rsync-mrtg[32246]: Total bytes sent: 18.12M
Jul 17 16:54:53 babylon-db-1 rsync-mrtg[32246]: Total bytes received: 4.68M
Jul 17 16:54:53 babylon-db-1 rsync-mrtg[32246]:
Jul 17 16:54:53 babylon-db-1 rsync-mrtg[32246]: sent 18.12M bytes
received 4.68M bytes  78.78K bytes/sec
Jul 17 16:54:53 babylon-db-1 rsync-mrtg[32246]: total size is 4.30G
speedup is 188.47
Jul 17 16:54:53 babylon-db-1 rsync-mrtg[32686]: Finished

So 3121 files totaling 4.3GB in size, and at least 90% of those files
would change between successive sync runs, yet only a very small
amount of data needs to be transferred.

I'd also add that in my experience rsync is incredibly robust and reliable.
I've been running an hourly rsync from my main MRTG server to 3
separate "slaves" for almost 2 years now, and never once had a problem
with data integrity.

Hope this helps,
Darren

On 17 July 2012 16:44, Simon Hobson <linux at thehobsons.co.uk> wrote:
> ccddtt wrote:
>
>>1\ rsync,is a good sync file tools. but i think is not for me question.
>>because: when i update some data to rrd file,the file is status is
>>'changed',so rsync need transfer the complete file .Is my analysis
>>correct?
>
> No, you don't understand how rsync works.
> The main feature of rsync is it's ability to copy parts of a file.
> This makes it very efficient at updating files across a low bandwidth
> and/or high latency link (it parallelises operations to deal with
> high latency).
>
> See http://en.wikipedia.org/wiki/Rsync and also in particular see
> external link #7 http://samba.org/~tridge/phd_thesis.pdf which is
> Andrew Tridgell's thesis on sorting from which rsync sprang.
>
> Short version: He worked on an efficient process for sorting, and
> from that comes an efficient process for working out which bits of a
> file have changed - sorting and searching a list are somewhat related.
> The basis of the rsync protocol is :
> 1) use the part of the process to make fast (sheap) checksums of
> blocks of the file and compare them to file blocks that are different.
> 2) Use the efficient search process to find if blocks have moved
> rather than been changed.
> 3) Copy only what's needed to re-assemble the original file at the
> destination using what's already there in the previous version.
>
> While not applicable to your needs, rsync can even deal with a file
> where a small amount has been inserted or deleted (step 2 above). Eg
> if you add a single byte to the front of a 1GB file, then rsync will
> detect this and not copy the whole GB.
>
> It is worth reading the man page carefully, this compression isn't
> the default for all transfer modes, nor is encryption using the SSH
> protocol.
> --
> Simon Hobson
>
> Visit http://www.magpiesnestpublishing.co.uk/ for books by acclaimed
> author Gladys Hobson. Novels - poetry - short stories - ideal as
> Christmas stocking fillers. Some available as e-books.
>
> _______________________________________________
> rrd-users mailing list
> rrd-users at lists.oetiker.ch
> https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users