22

I'm buying an external hard drive to back up the computers in my house (finally!!). I'm hoping to use rsync. I've seen an example that does (or seems to do) exactly what I want. Something like this:

rsync -aE --delete /path/to/what/I/want/to/backup /Volumes/FW200/Backups

However, when looking at the rsync documentation and examples and so on, things started to look MUCH more complicated than that. Networking and daemons and jargon, oh my!

I'm assuming that none of that stuff is necessary as long as I'm just rsyncing from a computer to a firewire-connected external drive. I'm I wrong in assuming that? Are things really going to be more complicated than that innocuous command?

1

12 Answers 12

12

Rsync works fine across local drives. However, if it detects local paths it automatically goes into --whole-file mode which does not copy the diffs, but just copies the source file over the destination file. Rsync will still ignore files that haven't changed at all though. When bandwidth between the source and destination is high (like two local disks) this is much faster than reading both files, then copying just the changed bits.

2
  • 2
    s/However/Fortunately/ when files are local, reading both and doing all the diff magic to write the new with parts of both (reading them again) is a lot more work than just reading source, writing destination.
    – Javier
    Jun 14 '09 at 13:26
  • didn't know if I should get into that in my answer. I'll add it now.
    – kbyrd
    Jun 14 '09 at 23:13
33

I use rsync with the following flags handily memorable as 'glop' and 'trunc' and 'v'.

rsync -gloptrunc $srcdir $dstdir

A brief guide:

  • g - preserve group ownership info
  • l - copy symlinks as symlinks
  • o - preserve owner info
  • p - preserve permissions
  • t - preserve timestamps
  • r - recurse thru directories
  • u - update, skip any newer files
  • [n] - no, dont do this, do a dry run instead
  • c - checksum, attempt checksums on file blocks when possible (*)
    note: on local filesystems, this get overridden and entire files are copied instead.
  • v - verbose

I always run the above to make sure it works, then remove the 'n' flag that once I'm happy with the results.

The key features of the above combinations:

  • I run it in BOTH directions between two (or more) servers, thus syncing in BOTH directions. You update whichever you feel is the master at the time.
  • It allows either to be the master, with the significant caveat that if you want to delete something, you must delete it on both to be sure it's really gone, else it comes back.

I use this to keep two machines in sync, or to keep to subdirs in sync (like backing up to a USB drive).

As one of the other posts stated earlier, the 'checksum' may actually be forced OFF if you are dealing with local drives.

In some rare instances, I've had to add additional parameters to account for changes in login accounts across remote machines, changing ports, and even specifying where 'rsync' lives on the remote host... but those are not directly applicable to your question.

6
  • 3
    Doesn't the -a flag to rsync do all the above? Jul 16 '09 at 12:43
  • 2
    I'm not a big fan of using 'meta' flags when I dont know for sure what they do. The manapage for rsync indicates that '-a' actually is -rlptgoD on my box, which skips 'u' and 'v' but includes 'D'. of those, it seems that, for me, the addition of 'u' is the only key difference.
    – ericslaw
    Jul 17 '09 at 14:13
  • How do I force it to mirror from source to destination (deleting anything in the destination that is not in the source), rather than copying bi-directionally? Nov 23 '18 at 22:50
  • 1
    Looks like you'd need to use rsync -gloptrunc --delete --delete-excluded $srcdir $dstdir. The --delete says to delete anything in the destination that is not in the source, and --delete-excluded says to also delete excluded files in the destination which would normally not be tracked or copied. If you use --delete-excluded, technically you don't need to also use --delete, but it's ok to use them both anyway. Details are in the man pages here: linux.die.net/man/1/rsync. Nov 23 '18 at 22:57
  • 1
    Note: for general rsync usage I've written this tutorial for myself now too, so I can reference it as needed: How to use rsync Sep 2 '19 at 0:01
4

None of it is necessary, you can use rsync without any daemons or any other kind of configuration JUST FINE!

Just use the rsync command and you are good to go.

2

Judging by the path in your rsync command, would I be right in thinking you're using Mac OS X?

Personally, I'd opt for using Time Machine (if you're using Leopard), or Carbon Copy Cloner (http://www.bombich.com/software/ccc.html) which uses rsync.

Much easier than trying to right your own script. One advantage is that Time Machine and CCC will both give you incremental backups.

2

The example you used looks like it will work just fine for backups.

One thing you might want to consider when using rsync however is to make use of the --link-dest option. This lets you keep multiple backups, but use hard links for any unchanged files, effectively making all backups take the space of an incremental. An example use would be:

rsync -aE --link-dest=/mnt/external_disk/backup_20090612 dir_to_backup \
    /mnt/external_disk/backup_20090613

This assumes you have a dated backup for June 12 and you want to create a new one on June 13. You might want to omit the -v option if you don't want a printout of every file.

1
  • 1
    rsnapshot is a backup system that uses link-dest as you describe to keep full version backups in a space efficent manner. Jul 16 '09 at 12:42
2

You didn't mention your Operating system. Going by the assumption that its *nix based OS, your command is good.

However, if one or both drives happen to be NTFS formatted, being accessed from *nix or even from within Windows using Mobaxterm/cygwin, then rsync incremental functionality wouldn't work well with rsync -a (archive flag)

If NTFS drive(s) is/are involved, you could use:

rsync -rvh --size-only --delete /path/to/what/I/want/to/backup /Volumes/FW200/Backups

Download Mobaxterm ssh client to have rsync functionality on Windows.

Here is more information on using rsync with NTFS drives

1

It really depends on whether you're running databases or not. Rsync will grab a snapshot of every file, and ignore any intervening writes. If you want to back up a database, you should look at setting up an ignore filter and running DB dump tools before the rsync.

1

Your command as written should work, however you might want to look at a program called rsnapshot which is built on top of rsync and keeps multiple versions of files so you can go back and look at things as they were last week or last month. The configuration is pretty easy and it is really good at space optimization so unless you have a lot of churn it doesn't take up much more space then a single backup.

0

I tried using rsync for backup but it ended up being a mess. rsync is better suited to 'synchronize' rather than to backup. And it goes forever to compare large files.

I researched a bit and tried several ones (basically testing everyone from apt-cache search backup in ubuntu).

Finally I have ended up with 'backup2l - low-maintenance backup/restore tool', it's easy. I like the way it manages planning and rotation (in levels). I run it whenever I have my USB external drive attached from the command line but you can also automate it.

0

Try dirvish to do the backup. http://www.dirvish.org/
It uses the hardlinks from rsync in the so-called vaults. You can keep as much of your older dumps as the USB disk can take. Or set it up in a an automated way.

Once you understand the idea of dirvish it more convient to use than rsync with all his options it self.

1
  • rsnapshot does similar to dirvish Jul 16 '09 at 12:42
0

I find a solution for Windows based computer: http://www.itefix.no/i2/node/10650

To copy D:\ partition to an external drive K:\

rsync -aE --delete --progress /cygdrive/d/* /cygdrive/k

0

I do not use rsync with local drives but Rsync is wonderful for sync, cloning, backup and restore of data between networked linux systems. A fantastic Linux network enabled tool worth spending time learning. Learn how to use rsync with hard links (--link-dest=) and life will be good.

I, however, have found rsync not as useful with local drives.

In the name of speed, Rsync, in my experience, changes automatically many operational parameters when rsync believes it has detected two local drives. Making matters worse. What is considered "local" from rsync's perspective can at times not really be that local. One example rsync sees a mounted SMB network share as a local drive. One can argue and be correct in explaining that in this case for rsync as a program instance the drives are all "local" but this misses the point.

The point is that scripts that operate as expected when used between a local and a remote drives do not work as expected when the same script is used where rsnyc sees the data paths as two local drives. Many rsync options seem changed or do not seem to work as expected when working with all local drives. File updates can slow to a crawl when one of the "local" drives is a networked SMB share or a large slower USB drive.

For example with "cwrsync -av /local/files/ /mountedSMBshare/files" and no -c (checksum) option where need for transfer should to be determined by file size and date with all local drives I see whole files copied between source and destination when the files have not even changed. This is not helpful behavior when one "drive" is a slower SMB networked share and the other a slow NTFS USB drive. A ssh into the SMB share server would be much better but this is not always possible and windows stuff, much hated, is part of everyday commercial life.

I would have preferred that rsync's operation was consistent regardless of the drives "location" and simply provide a --option for the user to invoke "local" operation when a speed advantage was seen as available and helpful. In my humble opinion this would be more consistent operation making rsync easier to use and more functional.

1
  • Welcome to Server Fault! Please consider breaking your answer into several paragraphs, it'll be much easier to read.
    – Glorfindel
    May 29 '20 at 21:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.