Rsync is a fast and extraordinarily versatile file copying tool. It can copy locally, to/from another host over any remote shell, or to/from a remote rsync daemon

learn more… | top users | synonyms

9
votes
8answers
14k views

How do you synchronise huge sparse files (VM disk images) between machines?

Is there a command, such as rsync, which can synchronise huge, sparse, files from one linux server to another? It is very important that the destination file remains sparse. It may be longer (but ...
9
votes
4answers
13k views

rsync is just hanging, building file list

[root@centos /]# rsync -av --exclude thumbs /storage root@xx.27.1.xx:/storage root@xx.27.1.xx's password: building file list ... been sitting for about an hour... it's 135GB of images and folders ...
9
votes
4answers
11k views

How to speed up rsync for small files

I'm trying to transfer thousands of small files from one server to another using the following command: rsync -zr --delete /home/user/ user@10.1.1.1::backup Currently the transfer takes a long time ...
9
votes
4answers
3k views

Handling renamed files or directories in rsync

I am working on an rsync script for directory replication. I have it syncing only new and modified files or directories but am not liking the fact that its copying renamed files or directories as a ...
9
votes
1answer
2k views

Why does rsync spawn multiple processes for me?

I am using the following cron statement to backup from one folder to another folder in the same machine: 19 21 * * * root rsync -ac --delete /source/folder /dest/folder When I use pstree, I see ...
9
votes
2answers
2k views

Can I make rsync output summary only?

Because I use rsync to backup a directory which is very big, containing many sub-dirs and files, so I dont want to see the "incremental file list", I want to know the summary in the end only. but if I ...
8
votes
3answers
4k views

Why don't people just use rsync to back up vmware guests?

If I am running a modern vmware ESXi system, I can drop in a statically linked rsync binary and rsync files to any destination over SSH. I'm trying to understand why most (all ?) backup of vmware ...
8
votes
5answers
9k views

How does RSYNC does incremental Backups

How does RSYNC knows which files are chnages and which are not. Does it logs its data anywhere in the file. Because I want to do incrrmental backups but first it will transfer all files. So my main ...
8
votes
3answers
10k views

rsync won't delete files on destination

I am trying to mirror a directory that changes over time to another directory. My problem is that rsync is not deleting files on destination if they aren't existing in source directory anymore. Here ...
8
votes
3answers
7k views

compare 2 directories and copy differences in a 3rd directory

Running ubuntu 12.04, I want to compare 2 directories, say folder1/ and folder2/ and copy any files that are different to folder3/. There are also nested files, so matching subdirectories should be ...
8
votes
4answers
4k views

Converting UTF-8 NFD filenames to UTF-8 NFC, in either rsync or afpd

I have a home file server running FreeNAS 8. A few days ago I used rsync to upload my entire iTunes library from Mac so that I could load my library over the network instead of off a slow USB drive. ...
8
votes
5answers
11k views

ZFS Sync over unreliable, slow WAN. ZFS replication, or rsync?

I've been tasked with making an off-site backup work over the WAN. Both storage boxes are FreeBSD based NAS boxes running ZFS. Once or twice a week, 15-60 gigs of photography data gets dumped to the ...
8
votes
2answers
10k views

How to repeatedly call rsync until files are sucessfully transferred

I'm trying to syncronize files from a remote server that is not reliable, meaning the connection tends to fail "randomly" with rsync: connection unexpectedly closed Rsync is called with ...
8
votes
5answers
3k views

Forcing rsync to non-interactive mode

I would like to use rsync within a python script. I'm calling it using the subprocess module, and authenticating using public keys stored at the authorized_key file on the remote machine. The only ...
8
votes
3answers
6k views

Encrypted Remote Backups via Rsync?

I run a small modest CentOS server at my house for misc backups, etc. My friend on the other side of town also runs a small modest server at his house for similar purposes. We have been toying with ...
8
votes
3answers
7k views

Maximizing rsync performance and throughput - directly-connected gigabit servers

I have two Dell R515 servers running CentOS 6.5, with one of the broadcom NICs in each directly attached to the other. I use the direct link to push backups from the main server in the pair to the ...
8
votes
2answers
7k views

Can't get rsync to work in daemon-over-ssh mode

I'm trying to set up rsync to copy the data from a server every day. In order to make the system as restricted as possible, I'm trying to use the mode described in the man page as: "USING RSYNC-DAEMON ...
8
votes
1answer
368 views

How to cleanup after failed rsync?

I killed a local "rsync -ah" process. It left junk files like ".abc.vmdk.zxivMN" in the target directory. The junk files do not get deleted on consecutive "rsync -ah" runs. Details: I ran diff on ...
8
votes
6answers
20k views

Is there any other reason for “no space left on device”?

I am using Dirvish on a Ubuntu server system for backing up a hd to an external usb 3.0 drive. Until a few days ago, everything worked fine, but now every backup fails with "no space left on device ...
8
votes
4answers
13k views

Multi Threaded file sync between 2 Linux servers

At the moment i'm running rsync for 2.2 million files total of 250GB and that just takes ages 700K files in 6 hours. Does anyone know a rsync like tool that can do this with multiple threads so it ...
8
votes
2answers
11k views

rsync copy new files only, no updates?

I would like to use rsync to copy files from a remote server. I would like to only copy files from server, not push any back out. copy files that don't exist. If the remote file is newer I do not ...
7
votes
8answers
3k views

make a cronjob wait for previous rsync job to finish

I'm using rsync to backup some data from one server to another. All works fine, but it might take longer to finish depending on how much data there is to transfer. Is there any gauranteed way to ...
7
votes
4answers
2k views

How can I easily confirm in Linux that two separate directories have the exact same contents?

CentOS 5.x Mq question seemed similar to this one but I wasn't sure... I have two servers (completely isolated from each other), each with a directory and sub-directories that should have the same ...
7
votes
9answers
1k views

Copy 10 million images in a single folder to another server

Now I know you shouldn't ever put 10 million files into a single directory to begin with. Blame it on the developers, but as it stands that’s where I am at. We will be fixing it and moving them into ...
7
votes
6answers
6k views

improving rsync backup performance

What are the best techniques to improve rsync over ssh mirroring between unix boxes, assuming that one system will always have the master copy and the other system will always have a recent copy (less ...
7
votes
2answers
1k views

How do I use rsync with an IPv6 address via SSH?

I've been having difficulty with connecting to my IPv6 address via rsync. Because the argument for the destination folder is colon-separated, the IPv6 address disrupts this like so: ...
7
votes
2answers
6k views

rsync : transferring files in one direction only

Is it possible to use rsync to copy files in one direction only? For example, suppose we have: left/a.txt right/a.txt where the files are initially identical. If one then modifies right/a.txt, ...
7
votes
3answers
4k views

rsync 'cannot delete non-empty directory' errors, even with --force option

When running this command: $ sudo rsync -r --delete --force --checksum --exclude=uploads /data/prep/* /data/app/ I'm getting the following output: cannot delete non-empty directory: ...
7
votes
4answers
5k views

Rsync show progress for individual file

I'm using rsync to upload/download a pretty large file (20Gb). I'm aware of --progress option, but what it does in my case is just showing me the result at the end. My problem is that during some ...
7
votes
4answers
1k views

linux to linux, 10TB transfer?

I've looked at all the previous similar questions, but the answers seemed to be all over the place and no one was moving a lot of data (100GB != 10TB). I've got about 10TB that I need to move from ...
7
votes
2answers
8k views

How to add timestamp and file list to rsync log?

I want to add both timestamp and file list to the log. Currently, I can only get one. The following command to can track the updated file list but it does not add timestamp. rsync -avz --progress ...
7
votes
4answers
5k views

rsync to compressed archive of file

I take backups of a server with rsync, but those backups are quite large, and I'd like to compress them. Is there a way to have some sort of wrapper around rsync to gunzip the file in the backup, ...
7
votes
1answer
8k views

What exactly will --delete-excluded do for rsync?

I use rsync with great pleasure for backing up my servers, although I do have a question about the --delete-excluded parameter. /usr/bin/rsync -a --delete --numeric-ids --relative --delete-excluded ...
7
votes
3answers
1k views

Tell Rsync to Skip Current file During Transfer?

Is there a way to tell rsync to skip its current file while a sync is in progress, maybe by sending it a particular signal? I already know about ignoring based on patterns, but this would be handy to ...
7
votes
1answer
5k views

How can I prevent rsync replacing symbolic links with new directories in the target directory hierarchy?

I am using rsync to synchronize files between a source area and a target area, where the target directory hierarchy contains symbolic links. When a source directory has the same name and relative ...
7
votes
2answers
2k views

Get list of transferred files from rsync?

I'm currently using rsync on a script that deploys a PHP application from a staging to a production server. Here is how: rsync -rzai --progress --stats --ignore-times --checksum /tmp/app_export/ ...
6
votes
4answers
2k views

How to copy a 200GB file faster?

I got a 200GB .tgz file on server A(RHEL 5.2). I wanna transfer that file to server B (RHEL 5.3). Server B is on ESXi 4 Update1. I gave 10GB to that Server B VM, with 4 vCPUs. Both Server A and ...
6
votes
10answers
21k views

Using Rsync to backup to an external drive

I'm buying an external hard drive to back up the computers in my house (finally!!). I'm hoping to use rsync. I've seen an example that does (or seems to do) exactly what I want. Something like ...
6
votes
4answers
2k views

Do you need -e ssh for rsync?

I always used to use the following command when copying from a server: rsync --progress -avze ssh user@host:/path/to/files ./here. However, a friend of mind showed me that I can simply do rsync ...
6
votes
4answers
5k views

How can I reduce resource usage when copying a large file?

I need to move a large file (corrupt MySQL table ~40GB) onto a seperate server in order to repair it. (When trying to repair on my production server, it quickly killed the server). In order to do ...
6
votes
3answers
13k views

run rsync as root but keep user ownership

I would like to backup user files from one server to another with rsync. but I noticed that the user folders change to root. how can I keep the user permissions with rsync (running by root)?
6
votes
2answers
4k views

rsync: How to exclude Dotfiles only in topmost directory?

When performing backups under Bash with rsync, I'm trying to exclude all dotfiles and hidden directories in the top directory, but not those in otherwise targeted directories. For example: /copyme.c ...
6
votes
3answers
8k views

Which is best for backups rsync vs rdiff vs rsnapshot

I am using rsync for backups from remote FTP to local computer. I read on internet that rsnapshot is better. Just want to know which is used in production environment
6
votes
2answers
2k views

Can rsync display current average speed?

When I'm transferring large quantities of data using rsync, it would be helpful if I could have the average speed up until now at a glance, rather than a bunch of different speeds for each file.
6
votes
3answers
7k views

Rsync seems incompatible with .bashrc (causes “is your shell clean?”)

It turns out rsync can't work with a remote server which has a .bashrc file? At local client i got when run rsync: protocol version mismatch -- is your shell clean? (see the rsync man page for an ...
6
votes
3answers
4k views

Rsync over ssh with root access on both sides

I have one older ubuntu server, and one newer debian server and I am migrating data from the old one to the new one. I want to use rsync to transfer data across to make final migration easier and ...
6
votes
8answers
3k views

rsyncing a bootable backup

jwz mentions PSA: Backups a backup system that's fairly clever. Your backup method is to essentially restore to the replacement drive. This seems to work simply on OSX. What all do I need to do to ...
6
votes
2answers
2k views

Copy directory tree without empty directories?

I have the following tree # upper letters = directory # lower letters = files A |-- B |-- C |-- D |-- e <= file |-- F |-- G I need to copy this tree to another destination, ...
6
votes
6answers
2k views

Remote file copy util (like rsync) but that will take account of data already copied (in this session)?

Let's say I have a directory with 2 files, both are identical and quite large (e.g. 2GB ea.) I want to rsync that directory to a remote host. As I understand it (and I could be wrong), rsync ...
6
votes
3answers
7k views

How to fix corrupt packet error for with rsync for (relatively) large files?

Trying to update files on a server, with the rsync command: rsync -ravq -e "ssh -o ConnectTimeout=2 -o ServerAliveInterval=2 -ServerAliveCountMax=2" --delete ./local_dir user@$SERVER:/dest_dir ...