Rsync is a fast and extraordinarily versatile file copying tool. It can copy locally, to/from another host over any remote shell, or to/from a remote rsync daemon

learn more… | top users | synonyms

90
votes
15answers
142k views

How to use rsync over FTP

Any unix: I have the following cmd line which works fine. rsync -avr -e ssh /home/dir user@example.com:/home/ But I need to set it up now to rsync to a remote server that only has an FTP server on ...
78
votes
15answers
81k views

Copying a large directory tree locally? cp or rsync?

I have to copy a large directory tree, about 1.8 TB. It's all local. Out of habit I'd use rsync, however I wonder if there's much point, and if I should rather use cp. I'm worried about permissions ...
43
votes
3answers
29k views

What is archive mode in rsync?

I know you can use -a or --archive to activate archive mode when using rsync. Unfortunately, I have no idea what archive mode is supposed to do, and the man page is not at all explicit about what this ...
41
votes
15answers
38k views

Showing total progress in rsync: is it possible?

I have searched for this option already, but have only found solutions that involve custom patching. The fact that it does not show in --help and no more info can be found probably indicates the ...
41
votes
15answers
9k views

Favorite rsync tips and tricks

The more I use rsync the more I realise that it's a swiss army knife of file transfer. There are so many options. I recently found out that you can go --remove-source-files and it'll delete a file ...
40
votes
18answers
38k views

Why hasn't rsync caught on in the Windows world? [closed]

It seems that rsync is the de-facto standard for efficient file backup and sync in Unix/Linux. Does anyone have any thoughts on why it wouldn't have caught on in the Windows world? Why hasn't it ...
25
votes
8answers
18k views

Why rsync is faster than NFS?

Few days ago I noticed something rather odd (at least for me). I ran rsync copying the same data and deleting it afterwards to NFS mount, called /nfs_mount/TEST. This /nfs_mount/TEST is ...
23
votes
7answers
5k views

Options to efficiently synchronize 1 million files with remote servers?

At a company I work for we have such a thing called "playlists" which are small files ~100-300 bytes each. There's about a million of them. About 100,000 of them get changed every hour. These ...
21
votes
6answers
19k views

Using RSYNC with Amazon S3

I am interested in using Amazon S3 to backup our ~ 100gb server images (created via Acronis backup tools) Obviously, this uploading to S3 every night would be expensive, in terms of bandwidth and ...
21
votes
3answers
23k views

Why is my rsync so slow?

My Laptop and my workstation are both connected to a Gigabit Switch. Both are running Linux. But when I copy files with rsync, it performs badly. I get about 22 MB/s. Shouldn't I theoretically get ...
20
votes
3answers
14k views

How to keep Rsync from chown'ing transfered files?

I have a sshfs connection setup with a remote filesystem on a Linux server. I'm doing an Rsync from my local server to the ftpfs-filesystem. Because of the nature of this setup, I can't chown anything ...
20
votes
5answers
21k views

Rsync difference between --checksum and --ignore-times options

Can anyone clarify the differences between the --checksum and --ignore-times options of rsync? My understanding is as follows: --checksum If the file size and time match, it will do a checksum at ...
19
votes
5answers
24k views

Is it possible to use rsync over sftp (without an ssh shell)?

Rsync over ssh, works great every time. However, trying to rsync to a host which allows only sftp logins, but not ssh logins, provides the following error: rsync -av /source ssh ...
19
votes
2answers
10k views

How to keep the full path with rsync?

When backing up with rsync, How do I keep the full directory structure? For example, the remote server is saturn, and I want to backup saturn's /home/udi/files/pictures to a local directory named ...
18
votes
9answers
18k views

Best way to compare (diff) a full directory structure?

What's the best way to compare directory structures? I have a backup utility which uses rsync. I want to tell the exact differences (in terms of file sizes and last-changed dates) between the source ...
16
votes
6answers
16k views

protocol version mismatch — is your shell clean?

When following the instructions to do rsync backups given here: http://troy.jdmz.net/rsync/index.html I get the error "protocol version mismatch -- is your shell clean?" I read somewhere that I ...
15
votes
9answers
5k views

Tool or script to detect moved or renamed files on Linux prior to a backup

Basically I am searching to see if there exists a tool or script that can detect moved or renamed files so that I can get a list of renamed/moved files and apply the same operation on the other end of ...
15
votes
2answers
946 views

Does Rsync Allow Files To Be Synced Both Ways?

I have backed up a linux web server using rsync with cygwin. I now have a perfect copy of the server on my windows laptop. If i delete or modify a file on my laptop and run rsync again with cygwin ...
14
votes
13answers
8k views

Complete restore of linux system

I am familiar with using rsync to back up various files from my system but what is the best way to completely restore a machine. What I have tried in the past is: Do a basic format/reinstall from ...
14
votes
5answers
5k views

Nice rsync on remote machine

When using rsync+ssh to access a remote machine, is there a way to "nice" the rsync process on the remote machine (to lower its priority)? Editing the question to clarify: USER PID %CPU %MEM ...
14
votes
3answers
18k views

Getting rsync to move file from source to destination?

Is rsync is a good choice for my project ? I have to : - copy files from source to destination folder via SSH, - be sure all files are copied, - delete source files after copy. - if I have conflict ...
13
votes
8answers
18k views

Sync LVM snapshots to backup server

I have a number of Xen virtual machines running on a number of Linux servers. These VMs store their disk images in Linux LVM volumes with device names along the lines of /dev/xenVG/SERVER001OS and so ...
13
votes
6answers
7k views

Unix copy command that has a progress bar, but not as heavy as rsync

I need to copy lots of files. Usually I use rsync because I pass it the -aP options and I can see (a) how many files are left to process and (b) how much of each individual file is copied. However ...
13
votes
1answer
8k views

rsync - Exclude files that are over a certain size?

I am doing a backup of my desktop to a remote machine. I'm basically doing rsync -a ~ example.com:backup/ However there are loads of large files, e.g. wikipedia dumps etc. Most of the files I care a ...
13
votes
7answers
9k views

How can I use rsync to duplicate a directory tree, creating hardlinks to files?

From time to time, I have to perform several large migration changes on data files on my server, and I'm looking for a good way to do this. I was thinking about using rsync to duplicate my directory ...
13
votes
9answers
11k views

rsync to multiple destinations using same filelist?

I'm wondering if it's possible for rsync to copy one directory to multiple remote destinations all in one go, or even in parallel. (not necessary, but would be useful.) Normally, something like the ...
13
votes
4answers
6k views

Rsync daemon: is it really useful?

Are there any practical benefits in using rsyncd compared to rsync over ssh? Does it really increase speed, stability, anything?
13
votes
9answers
9k views

Windows rsync that supports long file names, or an good alternative?

I currently rsync on a Linux host to copy things from Windows to my Linux box. But I can't copy files with long names. I have tried DeltaCopy, cwrsync, and cygwin. From what I have found there all ...
12
votes
1answer
17k views

Why is my rsync so slow compared to pure cp or even scp?

I'm transfering the files from Linux to Windows 7 via a mounted share (the share is mounted from Windows on Linux).. I'm copying lots of data (i.e. nearly a TB) from the old to the new machine within ...
12
votes
4answers
4k views

btrfs-enabled backup solution

With btrfs hitting production in Oracle EL 14th this month (together with working fsck and scrubbing from Linux 3.2) I was thinking of redesigning my current backup solution to utilise it. Note that ...
12
votes
2answers
803 views

Inode tables shrinking sharply over time causing rsync/inode problems

We setup a Linux (it's on Amazon AWS, a CentOS-like system although we are not exactly sure the customizations done on it) system with 4TB storage as an XFS volume over LVM (ultimately to be used for ...
10
votes
4answers
1k views

Deploying Files To Multiple Servers

We have a central repository of files in say server-1 at /srv/www. Then we have N number of servers as well. We want server-1 to be able to deploy its files in /srv/www to all servers as quickly and ...
10
votes
2answers
313 views

Server migration: most efficient way

I have been tasked with migrating one of our sites between servers (two different hosts). Both environments are linux. The site streams video, so the server is currently filled with media files ...
10
votes
2answers
12k views

Why is rsync skipping the main directory?

I'm trying to use rync locally (on a windows machine) to a remote server (my osx box) in order to test a remote deploy build script. I've done rsync before just fine between 2 linux servers, but I'm ...
10
votes
8answers
10k views

Copy large file from one Linux server to another

I'm attempting to copy a 75 gigabyte tgz (mysql lvm snapshot) from a Linux server in our LA data center to another Linux server in our NY data center over a 10MB link. I am getting about 20-30Kb/s ...
10
votes
6answers
6k views

Get rsync to generate a patch file instead of copying across files?

I'm copying lots of files that have changed from one server to another using rsync. I know I can use the -n option to do a dry run, so I can see what files have been changed. However is it possible to ...
10
votes
3answers
4k views

Rsync: remote source and destination [closed]

If both source and destination are remote, rsync complains: The source and destination cannot both be remote. rsync error: syntax or usage error (code 1) at main.c(1156) [Receiver=3.0.7] Is there a ...
9
votes
4answers
22k views

For large files compress first then transfer or rsync -z? which would be fastest?

I have a ton of relativity small data files but they take up about 50 GB and I need them transferred to a different machine. I was trying to think of the most efficient way to do this. Thoughts I had ...
9
votes
1answer
3k views

Getting rsync to delete files from the *sender* after transfer

I'm using rsync to move one directory to another. Is there some way to get rsync to incrementally delete a file from the sender after it has copied it to the reciever?
9
votes
5answers
9k views

Execute rsync command over ssh with an ssh agent via crontab

i have a cronjob: 0 9 * * * rsync -a mydir remote_machine: i installed this with 'crontab -e'. i have an ssh-agent running, and when i execute the rsync command itself it works w/o any user ...
9
votes
3answers
6k views

Has anyone achieved true differential sync with rsync in ESXi?

Berate me later on the fact that I'm using the service console to do anything in ESXi... I've got a working rsync binary (v3.0.4) that I can use in ESXi 4.1U1. I tend to use rsync over cp when ...
9
votes
3answers
970 views

How to mirror filesystems with millions of hardlinks?

We have one big problem at the moment: We need to mirror a filesystem for one of our customers. Thats usual not really a problem, but here it is: On this filesystem there is one folder with millions ...
9
votes
2answers
2k views

Which is better for website backup - rsync or git push

I run 2 LAMP web servers at different providers for disaster recovery purposes - a high powered live server, and a low powered backup server. Currently I rsync all the data from the live server to ...
8
votes
5answers
5k views

Rsync : copying over timestamps only

Currently I have two directories A/ and B/ which are identical in every respect, with the exception of the timestamps. Therefore if I run the command : rsync --dry-run -crvv A/ B/ then all files ...
8
votes
8answers
11k views

How do you synchronise huge sparse files (VM disk images) between machines?

Is there a command, such as rsync, which can synchronise huge, sparse, files from one linux server to another? It is very important that the destination file remains sparse. It may be longer (but ...
8
votes
5answers
6k views

How can I use rsync with a FAT file system?

I would like to write a simple backup script that saves some data to a FAT drive. Should I reformat the drive and use a better file system or is it possible to use rsync with FAT? If so, what problems ...
8
votes
4answers
9k views

ZFS Sync over unreliable, slow WAN. ZFS replication, or rsync?

I've been tasked with making an off-site backup work over the WAN. Both storage boxes are FreeBSD based NAS boxes running ZFS. Once or twice a week, 15-60 gigs of photography data gets dumped to the ...
8
votes
4answers
3k views

Synchronizing very large folder structures

We have a folder structure on our intranet which contains around 800,000 files divvied up into around 4,000 folders. We need to synchronize this to a small cluster of machines in our DMZs. The depth ...
8
votes
4answers
8k views

How to speed up rsync for small files

I'm trying to transfer thousands of small files from one server to another using the following command: rsync -zr --delete /home/user/ user@10.1.1.1::backup Currently the transfer takes a long time ...
8
votes
1answer
260 views

How to cleanup after failed rsync?

I killed a local "rsync -ah" process. It left junk files like ".abc.vmdk.zxivMN" in the target directory. The junk files do not get deleted on consecutive "rsync -ah" runs. Details: I ran diff on ...