Rsync is a fast and extraordinarily versatile file copying tool. It can copy locally, to/from another host over any remote shell, or to/from a remote rsync daemon

learn more… | top users | synonyms

12
votes
4answers
7k views

Converting UTF-8 NFD filenames to UTF-8 NFC, in either rsync or afpd

I have a home file server running FreeNAS 8. A few days ago I used rsync to upload my entire iTunes library from Mac so that I could load my library over the network instead of off a slow USB drive. ...
12
votes
3answers
11k views

Rsync seems incompatible with .bashrc (causes “is your shell clean?”)

It turns out rsync can't work with a remote server which has a .bashrc file? At local client i got when run rsync: protocol version mismatch -- is your shell clean? (see the rsync man page for an ...
12
votes
4answers
28k views

For large files compress first then transfer or rsync -z? which would be fastest?

I have a ton of relativity small data files but they take up about 50 GB and I need them transferred to a different machine. I was trying to think of the most efficient way to do this. Thoughts I had ...
12
votes
3answers
13k views

How to perform a secure rsync between servers across an unsecured network

Basically what I'm asking is, has anyone come across a means by which to wrap rsync inside ssh. With OpenSSH v4.9+ sftp has some nice options that allow you to chroot the incoming connection and such ...
12
votes
3answers
8k views

rsync 'cannot delete non-empty directory' errors, even with --force option

When running this command: $ sudo rsync -r --delete --force --checksum --exclude=uploads /data/prep/* /data/app/ I'm getting the following output: cannot delete non-empty directory: html/js/...
12
votes
1answer
4k views

Getting rsync to delete files from the *sender* after transfer

I'm using rsync to move one directory to another. Is there some way to get rsync to incrementally delete a file from the sender after it has copied it to the reciever?
12
votes
1answer
4k views

Why does rsync spawn multiple processes for me?

I am using the following cron statement to backup from one folder to another folder in the same machine: 19 21 * * * root rsync -ac --delete /source/folder /dest/folder When I use pstree, I see ...
12
votes
4answers
7k views

Rsync show progress for individual file

I'm using rsync to upload/download a pretty large file (20Gb). I'm aware of --progress option, but what it does in my case is just showing me the result at the end. My problem is that during some ...
12
votes
2answers
1k views

Inode tables shrinking sharply over time causing rsync/inode problems

We setup a Linux (it's on Amazon AWS, a CentOS-like system although we are not exactly sure the customizations done on it) system with 4TB storage as an XFS volume over LVM (ultimately to be used for ...
11
votes
8answers
19k views

How do you synchronise huge sparse files (VM disk images) between machines?

Is there a command, such as rsync, which can synchronise huge, sparse, files from one linux server to another? It is very important that the destination file remains sparse. It may be longer (but ...
11
votes
5answers
5k views

Synchronizing very large folder structures

We have a folder structure on our intranet which contains around 800,000 files divvied up into around 4,000 folders. We need to synchronize this to a small cluster of machines in our DMZs. The depth ...
11
votes
2answers
4k views

Which is better for website backup - rsync or git push

I run 2 LAMP web servers at different providers for disaster recovery purposes - a high powered live server, and a low powered backup server. Currently I rsync all the data from the live server to ...
10
votes
6answers
8k views

what is the fastest and most reliable way of transferring a lot of files?

I'm trying to transfer about 100k files totaling 90gb. Right now I'm using rsync daemon but its slow 3.4mb/s and I need to do this a number of times. I'm wondering what options do I have that would ...
10
votes
4answers
5k views

Do you need -e ssh for rsync?

I always used to use the following command when copying from a server: rsync --progress -avze ssh user@host:/path/to/files ./here. However, a friend of mind showed me that I can simply do rsync --...
10
votes
4answers
3k views

Deploying Files To Multiple Servers

We have a central repository of files in say server-1 at /srv/www. Then we have N number of servers as well. We want server-1 to be able to deploy its files in /srv/www to all servers as quickly and ...
10
votes
2answers
2k views

How do I use rsync with an IPv6 address via SSH?

I've been having difficulty with connecting to my IPv6 address via rsync. Because the argument for the destination folder is colon-separated, the IPv6 address disrupts this like so: root@fdff::ffff:...
10
votes
2answers
332 views

Server migration: most efficient way

I have been tasked with migrating one of our sites between servers (two different hosts). Both environments are linux. The site streams video, so the server is currently filled with media files (...
10
votes
5answers
15k views

How to speed up rsync for small files

I'm trying to transfer thousands of small files from one server to another using the following command: rsync -zr --delete /home/user/ user@10.1.1.1::backup Currently the transfer takes a long time (...
10
votes
3answers
7k views

Has anyone achieved true differential sync with rsync in ESXi?

Berate me later on the fact that I'm using the service console to do anything in ESXi... I've got a working rsync binary (v3.0.4) that I can use in ESXi 4.1U1. I tend to use rsync over cp when ...
10
votes
2answers
12k views

rsync copy new files only, no updates?

I would like to use rsync to copy files from a remote server. I would like to only copy files from server, not push any back out. copy files that don't exist. If the remote file is newer I do not ...
10
votes
3answers
1k views

How to mirror filesystems with millions of hardlinks?

We have one big problem at the moment: We need to mirror a filesystem for one of our customers. Thats usual not really a problem, but here it is: On this filesystem there is one folder with millions ...
9
votes
3answers
24k views

Rsync command issues, owner and group permissions doesn´t change

I´m trying to set owner and group via rsync and it doesn't seem to be working. This is the command: sudo rsync -rlptDvz --owner=cmsseren --group=cmsseren /home/serena/public_html/ -e ssh root@ip:/...
9
votes
2answers
10k views

rsync : transferring files in one direction only

Is it possible to use rsync to copy files in one direction only? For example, suppose we have: left/a.txt right/a.txt where the files are initially identical. If one then modifies right/a.txt, ...
9
votes
4answers
2k views

Linux to Linux, 10 TB transfer?

I've looked at all the previous similar questions, but the answers seemed to be all over the place and no one was moving a lot of data (100 GB is different from 10 TB). I've got about 10&...
9
votes
2answers
13k views

Can I use rsync to create a list of only changed files?

I am using rsync in a bash script to keep files in sync between a few servers and a NAS. One issue I have run into is trying to generate a list of the files that have changed from the during the rsync....
9
votes
3answers
12k views

How to add timestamp and file list to rsync log?

I want to add both timestamp and file list to the log. Currently, I can only get one. The following command to can track the updated file list but it does not add timestamp. rsync -avz --progress --...
9
votes
2answers
4k views

Copy directory tree without empty directories?

I have the following tree # upper letters = directory # lower letters = files A |-- B |-- C |-- D |-- e <= file |-- F |-- G I need to copy this tree to another destination, ...
9
votes
1answer
19k views

Optimizing rsync for transferring huge files quickly

This is what I need: Rsync daemon running on source server. Destination server does a rsync pull as below: rsync -ptv rsync://source_ip:document/source_path/*.abc destination_path/ Huge binary files ...
9
votes
1answer
12k views

What exactly will --delete-excluded do for rsync?

I use rsync with great pleasure for backing up my servers, although I do have a question about the --delete-excluded parameter. /usr/bin/rsync -a --delete --numeric-ids --relative --delete-excluded ...
9
votes
4answers
7k views

rsync to compressed archive of file

I take backups of a server with rsync, but those backups are quite large, and I'd like to compress them. Is there a way to have some sort of wrapper around rsync to gunzip the file in the backup, ...
8
votes
3answers
7k views

Why don't people just use rsync to back up vmware guests?

If I am running a modern vmware ESXi system, I can drop in a statically linked rsync binary and rsync files to any destination over SSH. I'm trying to understand why most (all ?) backup of vmware ...
8
votes
10answers
30k views

Using Rsync to backup to an external drive

I'm buying an external hard drive to back up the computers in my house (finally!!). I'm hoping to use rsync. I've seen an example that does (or seems to do) exactly what I want. Something like this:...
8
votes
5answers
12k views

How does RSYNC does incremental Backups

How does RSYNC knows which files are chnages and which are not. Does it logs its data anywhere in the file. Because I want to do incrrmental backups but first it will transfer all files. So my main ...
8
votes
5answers
13k views

ZFS Sync over unreliable, slow WAN. ZFS replication, or rsync?

I've been tasked with making an off-site backup work over the WAN. Both storage boxes are FreeBSD based NAS boxes running ZFS. Once or twice a week, 15-60 gigs of photography data gets dumped to the ...
8
votes
4answers
7k views

How can I reduce resource usage when copying a large file?

I need to move a large file (corrupt MySQL table ~40GB) onto a seperate server in order to repair it. (When trying to repair on my production server, it quickly killed the server). In order to do ...
8
votes
1answer
6k views

Does rsync delete local files if they were deleted on source folder?

Here's what I'm trying to do: In essence, get files from one net storage device synced to another net storage device. The solution was to rsync from the source net storage to a local machine. Then ...
8
votes
2answers
9k views

Can't get rsync to work in daemon-over-ssh mode

I'm trying to set up rsync to copy the data from a server every day. In order to make the system as restricted as possible, I'm trying to use the mode described in the man page as: "USING RSYNC-DAEMON ...
8
votes
4answers
7k views

Encrypted Remote Backups via Rsync?

I run a small modest CentOS server at my house for misc backups, etc. My friend on the other side of town also runs a small modest server at his house for similar purposes. We have been toying with ...
8
votes
1answer
628 views

How to cleanup after failed rsync?

I killed a local "rsync -ah" process. It left junk files like ".abc.vmdk.zxivMN" in the target directory. The junk files do not get deleted on consecutive "rsync -ah" runs. Details: I ran diff on ...
8
votes
6answers
34k views

Is there any other reason for “no space left on device”?

I am using Dirvish on a Ubuntu server system for backing up a hd to an external usb 3.0 drive. Until a few days ago, everything worked fine, but now every backup fails with "no space left on device (...
8
votes
4answers
15k views

Multi Threaded file sync between 2 Linux servers

At the moment i'm running rsync for 2.2 million files total of 250GB and that just takes ages 700K files in 6 hours. Does anyone know a rsync like tool that can do this with multiple threads so it ...
8
votes
3answers
13k views

How to fix corrupt packet error for with rsync for (relatively) large files?

Trying to update files on a server, with the rsync command: rsync -ravq -e "ssh -o ConnectTimeout=2 -o ServerAliveInterval=2 -ServerAliveCountMax=2" --delete ./local_dir user@$SERVER:/dest_dir ...
8
votes
4answers
7k views

Rsync -avzHP follows hardlinks instead of copying them as hardlinks

I use rsnapshot to create hourly/daily/weekly/monthly backups of my "work"-share. Now I'm trying to copy the whole backup-directory onto an external drive using rsync. I used this command/parameters ...
8
votes
1answer
6k views

How can I prevent rsync replacing symbolic links with new directories in the target directory hierarchy?

I am using rsync to synchronize files between a source area and a target area, where the target directory hierarchy contains symbolic links. When a source directory has the same name and relative ...
8
votes
3answers
4k views

Get list of transferred files from rsync?

I'm currently using rsync on a script that deploys a PHP application from a staging to a production server. Here is how: rsync -rzai --progress --stats --ignore-times --checksum /tmp/app_export/ root@...
7
votes
8answers
5k views

make a cronjob wait for previous rsync job to finish

I'm using rsync to backup some data from one server to another. All works fine, but it might take longer to finish depending on how much data there is to transfer. Is there any gauranteed way to ...
7
votes
7answers
16k views

Copying huge files between two remote machines - Efficiently

I have a shell script which keeps on copying huge files (2 GB to 5 GB) between remote systems. Key based authentication is used with agent-forwarding and everything works. For ex: Say the shell script ...
7
votes
6answers
8k views

How do I copy a directory tree but not the files in Linux?

I want to copy about 200 directories & subdirectories from one location to another but I don't want to copy the thousands of files within those directories. I am on Linux. Note: I don't have ...
7
votes
6answers
15k views

Rsync over ssh: “ERROR: module is read only” suddenly appeared

I've used from some time rsync/ssh to backup my shared host contents to my personal Synology NAS (212j for that matter), and it worked quite well. For information, I use a password-less ssh ...
7
votes
4answers
2k views

How can I easily confirm in Linux that two separate directories have the exact same contents?

CentOS 5.x Mq question seemed similar to this one but I wasn't sure... I have two servers (completely isolated from each other), each with a directory and sub-directories that should have the same ...