Rsync is a fast and extraordinarily versatile file copying tool. It can copy locally, to/from another host over any remote shell, or to/from a remote rsync daemon

learn more… | top users | synonyms

138
votes
13answers
152k views

Copying a large directory tree locally? cp or rsync?

I have to copy a large directory tree, about 1.8 TB. It's all local. Out of habit I'd use rsync, however I wonder if there's much point, and if I should rather use cp. I'm worried about permissions ...
135
votes
16answers
204k views

How to use rsync over FTP

Any unix: I have the following cmd line which works fine. rsync -avr -e ssh /home/dir user@example.com:/home/ But I need to set it up now to rsync to a remote server that only has an FTP server on ...
97
votes
16answers
89k views

Showing total progress in rsync: is it possible?

I have searched for this option already, but have only found solutions that involve custom patching. The fact that it does not show in --help and no more info can be found probably indicates the ...
81
votes
3answers
51k views

What is archive mode in rsync?

I know you can use -a or --archive to activate archive mode when using rsync. Unfortunately, I have no idea what archive mode is supposed to do, and the man page is not at all explicit about what this ...
63
votes
18answers
71k views

Why hasn't rsync caught on in the Windows world? [closed]

It seems that rsync is the de-facto standard for efficient file backup and sync in Unix/Linux. Does anyone have any thoughts on why it wouldn't have caught on in the Windows world? Why hasn't it ...
62
votes
3answers
4k views

Rsync triggered Linux OOM killer on a single 50 GB file

I have a single 50 GB file on server_A, and I'm copying it to server_B. I run server_A$ rsync --partial --progress --inplace --append-verify 50GB_file root@server_B:50GB_file Server_B has 32 ...
47
votes
16answers
14k views

Favorite rsync tips and tricks

The more I use rsync the more I realise that it's a swiss army knife of file transfer. There are so many options. I recently found out that you can go --remove-source-files and it'll delete a file ...
41
votes
2answers
19k views

How to keep the full path with rsync?

When backing up with rsync, How do I keep the full directory structure? For example, the remote server is saturn, and I want to backup saturn's /home/udi/files/pictures to a local directory named ...
39
votes
5answers
50k views

Rsync difference between --checksum and --ignore-times options

Can anyone clarify the differences between the --checksum and --ignore-times options of rsync? My understanding is as follows: --checksum If the file size and time match, it will do a checksum at ...
36
votes
3answers
35k views

How to keep Rsync from chown'ing transfered files?

I have a sshfs connection setup with a remote filesystem on a Linux server. I'm doing an Rsync from my local server to the ftpfs-filesystem. Because of the nature of this setup, I can't chown anything ...
35
votes
5answers
43k views

Is it possible to use rsync over sftp (without an ssh shell)?

Rsync over ssh, works great every time. However, trying to rsync to a host which allows only sftp logins, but not ssh logins, provides the following error: rsync -av /source ssh user@remotehost:/...
32
votes
6answers
32k views

protocol version mismatch — is your shell clean?

When following the instructions to do rsync backups given here: http://troy.jdmz.net/rsync/index.html I get the error "protocol version mismatch -- is your shell clean?" I read somewhere that I ...
29
votes
2answers
13k views

Rsync creates a directory with the same name inside of destination directory

When I run this command rsync -avzp --del -e "ssh -p myport" user@hostname:/var/www/tests /var/www/tests files get synchronized but instead of saving files in /var/www/tests, Rsync creates one more ...
29
votes
8answers
26k views

Why rsync is faster than NFS?

Few days ago I noticed something rather odd (at least for me). I ran rsync copying the same data and deleting it afterwards to NFS mount, called /nfs_mount/TEST. This /nfs_mount/TEST is hosted/...
28
votes
2answers
21k views

Perform rsync while following sym links

I have a directory that contains symbolic links to other directories located on different media on my system: /opt/lun1/2011 /opt/lun1/2010 /opt/lun2/2009 /opt/lun2/2008 /opt/lun3/2007 But the ...
28
votes
3answers
47k views

Why is my rsync so slow?

My Laptop and my workstation are both connected to a Gigabit Switch. Both are running Linux. But when I copy files with rsync, it performs badly. I get about 22 MB/s. Shouldn't I theoretically get ...
27
votes
1answer
15k views

rsync - Exclude files that are over a certain size?

I am doing a backup of my desktop to a remote machine. I'm basically doing rsync -a ~ example.com:backup/ However there are loads of large files, e.g. wikipedia dumps etc. Most of the files I care a ...
26
votes
7answers
8k views

Options to efficiently synchronize 1 million files with remote servers?

At a company I work for we have such a thing called "playlists" which are small files ~100-300 bytes each. There's about a million of them. About 100,000 of them get changed every hour. These ...
25
votes
9answers
26k views

Best way to compare (diff) a full directory structure?

What's the best way to compare directory structures? I have a backup utility which uses rsync. I want to tell the exact differences (in terms of file sizes and last-changed dates) between the source ...
24
votes
7answers
24k views

How can one efficiently use S3 to back up files incrementally?

I understand how rsync works on a high-level, but there are 2 sides. With S3 there is no daemon to speak of — well there is, but it's basically just HTTP. There look to be a few approaches. s3rsync ...
24
votes
7answers
32k views

Using RSYNC with Amazon S3

I am interested in using Amazon S3 to backup our ~ 100gb server images (created via Acronis backup tools) Obviously, this uploading to S3 every night would be expensive, in terms of bandwidth and ...
24
votes
2answers
34k views

Why is rsync skipping the main directory?

I'm trying to use rync locally (on a windows machine) to a remote server (my osx box) in order to test a remote deploy build script. I've done rsync before just fine between 2 linux servers, but I'm ...
21
votes
5answers
13k views

Rsync : copying over timestamps only

Currently I have two directories A/ and B/ which are identical in every respect, with the exception of the timestamps. Therefore if I run the command : rsync --dry-run -crvv A/ B/ then all files ...
20
votes
6answers
7k views

Nice rsync on remote machine

When using rsync+ssh to access a remote machine, is there a way to "nice" the rsync process on the remote machine (to lower its priority)? Editing the question to clarify: USER PID %CPU %MEM ...
18
votes
8answers
13k views

How can I use rsync to duplicate a directory tree, creating hardlinks to files?

From time to time, I have to perform several large migration changes on data files on my server, and I'm looking for a good way to do this. I was thinking about using rsync to duplicate my directory ...
18
votes
13answers
11k views

Complete restore of linux system

I am familiar with using rsync to back up various files from my system but what is the best way to completely restore a machine. What I have tried in the past is: Do a basic format/reinstall from ...
18
votes
4answers
10k views

Rsync daemon: is it really useful?

Are there any practical benefits in using rsyncd compared to rsync over ssh? Does it really increase speed, stability, anything?
18
votes
3answers
8k views

Does Rsync Allow Files To Be Synced Both Ways?

I have backed up a linux web server using rsync with cygwin. I now have a perfect copy of the server on my windows laptop. If i delete or modify a file on my laptop and run rsync again with cygwin ...
17
votes
9answers
24k views

Sync LVM snapshots to backup server

I have a number of Xen virtual machines running on a number of Linux servers. These VMs store their disk images in Linux LVM volumes with device names along the lines of /dev/xenVG/SERVER001OS and so ...
17
votes
2answers
14k views

How to repeatedly call rsync until files are sucessfully transferred

I'm trying to syncronize files from a remote server that is not reliable, meaning the connection tends to fail "randomly" with rsync: connection unexpectedly closed Rsync is called with --...
17
votes
3answers
13k views

compare 2 directories and copy differences in a 3rd directory

Running ubuntu 12.04, I want to compare 2 directories, say folder1/ and folder2/ and copy any files that are different to folder3/. There are also nested files, so matching subdirectories should be ...
17
votes
3answers
8k views

Why it's not possible to use two remotes for rsync? [closed]

When both source and destination are remote, rsync complains: The source and destination cannot both be remote. rsync error: syntax or usage error (code 1) at main.c(1156) [Receiver=3.0.7] Is there ...
16
votes
5answers
11k views

How can I use rsync with a FAT file system?

I would like to write a simple backup script that saves some data to a FAT drive. Should I reformat the drive and use a better file system or is it possible to use rsync with FAT? If so, what problems ...
16
votes
3answers
17k views

Maximizing rsync performance and throughput - directly-connected gigabit servers

I have two Dell R515 servers running CentOS 6.5, with one of the broadcom NICs in each directly attached to the other. I use the direct link to push backups from the main server in the pair to the ...
15
votes
3answers
17k views

rsync won't delete files on destination

I am trying to mirror a directory that changes over time to another directory. My problem is that rsync is not deleting files on destination if they aren't existing in source directory anymore. Here ...
15
votes
2answers
4k views

Can I make rsync output summary only?

Because I use rsync to backup a directory which is very big, containing many sub-dirs and files, so I dont want to see the "incremental file list", I want to know the summary in the end only. but if I ...
15
votes
9answers
20k views

rsync to multiple destinations using same filelist?

I'm wondering if it's possible for rsync to copy one directory to multiple remote destinations all in one go, or even in parallel. (not necessary, but would be useful.) Normally, something like the ...
14
votes
3answers
24k views

run rsync as root but keep user ownership

I would like to backup user files from one server to another with rsync. but I noticed that the user folders change to root. how can I keep the user permissions with rsync (running by root)?
14
votes
4answers
18k views

rsync is just hanging, building file list

[root@centos /]# rsync -av --exclude thumbs /storage root@xx.27.1.xx:/storage root@xx.27.1.xx's password: building file list ... been sitting for about an hour... it's 135GB of images and folders /...
14
votes
8answers
15k views

Copy large file from one Linux server to another

I'm attempting to copy a 75 gigabyte tgz (mysql lvm snapshot) from a Linux server in our LA data center to another Linux server in our NY data center over a 10MB link. I am getting about 20-30Kb/s ...
14
votes
3answers
19k views

Getting rsync to move file from source to destination?

Is rsync is a good choice for my project ? I have to : - copy files from source to destination folder via SSH, - be sure all files are copied, - delete source files after copy. - if I have conflict ...
13
votes
6answers
8k views

Unix copy command that has a progress bar, but not as heavy as rsync

I need to copy lots of files. Usually I use rsync because I pass it the -aP options and I can see (a) how many files are left to process and (b) how much of each individual file is copied. However ...
13
votes
5answers
13k views

Execute rsync command over ssh with an ssh agent via crontab

i have a cronjob: 0 9 * * * rsync -a mydir remote_machine: i installed this with 'crontab -e'. i have an ssh-agent running, and when i execute the rsync command itself it works w/o any user ...
13
votes
2answers
6k views

Can rsync display current average speed?

When I'm transferring large quantities of data using rsync, it would be helpful if I could have the average speed up until now at a glance, rather than a bunch of different speeds for each file.
13
votes
4answers
7k views

Handling renamed files or directories in rsync

I am working on an rsync script for directory replication. I have it syncing only new and modified files or directories but am not liking the fact that its copying renamed files or directories as a ...
13
votes
1answer
19k views

Why is my rsync so slow compared to pure cp or even scp?

I'm transfering the files from Linux to Windows 7 via a mounted share (the share is mounted from Windows on Linux).. I'm copying lots of data (i.e. nearly a TB) from the old to the new machine within ...
13
votes
9answers
11k views

Windows rsync that supports long file names, or an good alternative?

I currently rsync on a Linux host to copy things from Windows to my Linux box. But I can't copy files with long names. I have tried DeltaCopy, cwrsync, and cygwin. From what I have found there all ...
13
votes
5answers
6k views

btrfs-enabled backup solution

With btrfs hitting production in Oracle EL 14th this month (together with working fsck and scrubbing from Linux 3.2) I was thinking of redesigning my current backup solution to utilise it. Note that I'...
13
votes
6answers
9k views

Get rsync to generate a patch file instead of copying across files?

I'm copying lots of files that have changed from one server to another using rsync. I know I can use the -n option to do a dry run, so I can see what files have been changed. However is it possible to ...
12
votes
2answers
17k views

Using rsync's archive flag without copying symbolic links

As stated in rsync's man page, the -a (archive) switch is equivalent to -rlptgoD. However, i have a situation where i don't want symbolic links retained. Is there any way to keep using the -a switch ...