Rsync is a fast and extraordinarily versatile file copying tool. It can copy locally, to/from another host over any remote shell, or to/from a remote rsync daemon

learn more… | top users | synonyms

8
votes
4answers
9k views

How to speed up rsync for small files

I'm trying to transfer thousands of small files from one server to another using the following command: rsync -zr --delete /home/user/ user@10.1.1.1::backup Currently the transfer takes a long time ...
8
votes
1answer
284 views

How to cleanup after failed rsync?

I killed a local "rsync -ah" process. It left junk files like ".abc.vmdk.zxivMN" in the target directory. The junk files do not get deleted on consecutive "rsync -ah" runs. Details: I ran diff on ...
8
votes
4answers
12k views

Multi Threaded file sync between 2 Linux servers

At the moment i'm running rsync for 2.2 million files total of 250GB and that just takes ages 700K files in 6 hours. Does anyone know a rsync like tool that can do this with multiple threads so it ...
8
votes
2answers
10k views

rsync copy new files only, no updates?

I would like to use rsync to copy files from a remote server. I would like to only copy files from server, not push any back out. copy files that don't exist. If the remote file is newer I do not ...
8
votes
5answers
13k views

Is there any other reason for “no space left on device”?

I am using Dirvish on a Ubuntu server system for backing up a hd to an external usb 3.0 drive. Until a few days ago, everything worked fine, but now every backup fails with "no space left on device ...
7
votes
3answers
2k views

Why don't people just use rsync to back up vmware guests?

If I am running a modern vmware ESXi system, I can drop in a statically linked rsync binary and rsync files to any destination over SSH. I'm trying to understand why most (all ?) backup of vmware ...
7
votes
8answers
2k views

make a cronjob wait for previous rsync job to finish

I'm using rsync to backup some data from one server to another. All works fine, but it might take longer to finish depending on how much data there is to transfer. Is there any gauranteed way to ...
7
votes
4answers
2k views

How can I easily confirm in Linux that two separate directories have the exact same contents?

CentOS 5.x Mq question seemed similar to this one but I wasn't sure... I have two servers (completely isolated from each other), each with a directory and sub-directories that should have the same ...
7
votes
6answers
5k views

what is the fastest and most reliable way of transferring a lot of files?

I'm trying to transfer about 100k files totaling 90gb. Right now I'm using rsync daemon but its slow 3.4mb/s and I need to do this a number of times. I'm wondering what options do I have that would ...
7
votes
9answers
1k views

Copy 10 million images in a single folder to another server

Now I know you shouldn't ever put 10 million files into a single directory to begin with. Blame it on the developers, but as it stands that’s where I am at. We will be fixing it and moving them into ...
7
votes
2answers
7k views

Using rsync's archive flag without copying symbolic links

As stated in rsync's man page, the -a (archive) switch is equivalent to -rlptgoD. However, i have a situation where i don't want symbolic links retained. Is there any way to keep using the -a switch ...
7
votes
6answers
6k views

improving rsync backup performance

What are the best techniques to improve rsync over ssh mirroring between unix boxes, assuming that one system will always have the master copy and the other system will always have a recent copy (less ...
7
votes
3answers
4k views

compare 2 directories and copy differences in a 3rd directory

Running ubuntu 12.04, I want to compare 2 directories, say folder1/ and folder2/ and copy any files that are different to folder3/. There are also nested files, so matching subdirectories should be ...
7
votes
3answers
7k views

rsync won't delete files on destination

I am trying to mirror a directory that changes over time to another directory. My problem is that rsync is not deleting files on destination if they aren't existing in source directory anymore. Here ...
7
votes
2answers
1k views

How do I use rsync with an IPv6 address via SSH?

I've been having difficulty with connecting to my IPv6 address via rsync. Because the argument for the destination folder is colon-separated, the IPv6 address disrupts this like so: ...
7
votes
4answers
11k views

rsync is just hanging, building file list

[root@centos /]# rsync -av --exclude thumbs /storage root@xx.27.1.xx:/storage root@xx.27.1.xx's password: building file list ... been sitting for about an hour... it's 135GB of images and folders ...
7
votes
5answers
9k views

How can one efficiently use S3 to back up files incrementally?

I understand how rsync works on a high-level, but there are 2 sides. With S3 there is no daemon to speak of — well there is, but it's basically just HTTP. There look to be a few approaches. s3rsync ...
7
votes
4answers
980 views

linux to linux, 10TB transfer?

I've looked at all the previous similar questions, but the answers seemed to be all over the place and no one was moving a lot of data (100GB != 10TB). I've got about 10TB that I need to move from ...
7
votes
3answers
5k views

Encrypted Remote Backups via Rsync?

I run a small modest CentOS server at my house for misc backups, etc. My friend on the other side of town also runs a small modest server at his house for similar purposes. We have been toying with ...
7
votes
4answers
3k views

Rsync show progress for individual file

I'm using rsync to upload/download a pretty large file (20Gb). I'm aware of --progress option, but what it does in my case is just showing me the result at the end. My problem is that during some ...
7
votes
3answers
2k views

Maximizing rsync performance and throughput - directly-connected gigabit servers

I have two Dell R515 servers running CentOS 6.5, with one of the broadcom NICs in each directly attached to the other. I use the direct link to push backups from the main server in the pair to the ...
7
votes
1answer
5k views

How can I prevent rsync replacing symbolic links with new directories in the target directory hierarchy?

I am using rsync to synchronize files between a source area and a target area, where the target directory hierarchy contains symbolic links. When a source directory has the same name and relative ...
6
votes
4answers
1k views

How to copy a 200GB file faster?

I got a 200GB .tgz file on server A(RHEL 5.2). I wanna transfer that file to server B (RHEL 5.3). Server B is on ESXi 4 Update1. I gave 10GB to that Server B VM, with 4 vCPUs. Both Server A and ...
6
votes
5answers
7k views

How does RSYNC does incremental Backups

How does RSYNC knows which files are chnages and which are not. Does it logs its data anywhere in the file. Because I want to do incrrmental backups but first it will transfer all files. So my main ...
6
votes
4answers
3k views

Converting UTF-8 NFD filenames to UTF-8 NFC, in either rsync or afpd

I have a home file server running FreeNAS 8. A few days ago I used rsync to upload my entire iTunes library from Mac so that I could load my library over the network instead of off a slow USB drive. ...
6
votes
2answers
8k views

How to repeatedly call rsync until files are sucessfully transferred

I'm trying to syncronize files from a remote server that is not reliable, meaning the connection tends to fail "randomly" with rsync: connection unexpectedly closed Rsync is called with ...
6
votes
4answers
2k views

Do you need -e ssh for rsync?

I always used to use the following command when copying from a server: rsync --progress -avze ssh user@host:/path/to/files ./here. However, a friend of mind showed me that I can simply do rsync ...
6
votes
4answers
4k views

How can I reduce resource usage when copying a large file?

I need to move a large file (corrupt MySQL table ~40GB) onto a seperate server in order to repair it. (When trying to repair on my production server, it quickly killed the server). In order to do ...
6
votes
5answers
3k views

Forcing rsync to non-interactive mode

I would like to use rsync within a python script. I'm calling it using the subprocess module, and authenticating using public keys stored at the authorized_key file on the remote machine. The only ...
6
votes
2answers
3k views

rsync: How to exclude Dotfiles only in topmost directory?

When performing backups under Bash with rsync, I'm trying to exclude all dotfiles and hidden directories in the top directory, but not those in otherwise targeted directories. For example: /copyme.c ...
6
votes
3answers
5k views

Rsync seems incompatible with .bashrc (causes “is your shell clean?”)

It turns out rsync can't work with a remote server which has a .bashrc file? At local client i got when run rsync: protocol version mismatch -- is your shell clean? (see the rsync man page for an ...
6
votes
2answers
6k views

Can't get rsync to work in daemon-over-ssh mode

I'm trying to set up rsync to copy the data from a server every day. In order to make the system as restricted as possible, I'm trying to use the mode described in the man page as: "USING RSYNC-DAEMON ...
6
votes
8answers
3k views

rsyncing a bootable backup

jwz mentions PSA: Backups a backup system that's fairly clever. Your backup method is to essentially restore to the replacement drive. This seems to work simply on OSX. What all do I need to do to ...
6
votes
3answers
1k views

Tell Rsync to Skip Current file During Transfer?

Is there a way to tell rsync to skip its current file while a sync is in progress, maybe by sending it a particular signal? I already know about ignoring based on patterns, but this would be handy to ...
6
votes
6answers
1k views

Remote file copy util (like rsync) but that will take account of data already copied (in this session)?

Let's say I have a directory with 2 files, both are identical and quite large (e.g. 2GB ea.) I want to rsync that directory to a remote host. As I understand it (and I could be wrong), rsync ...
6
votes
2answers
406 views

rsync for backup considered dangerous?

In our research group, we have the need to backup data acquired on the MRI scanner in a way that will preserve any scan ever acquired (even though data might be deleted from the scanner due to space ...
6
votes
2answers
2k views

Cron and rsync using file lock but is there a better way on Mac OS X Server?

I have this bash script and is run via cron on a regular interval: #!/bin/bash RSYNC=/usr/bin/rsync SSH=/usr/bin/ssh KEY=/Users/admin/Documents/Backup/rsync-key RUSER=philosophy RHOST=example.com ...
5
votes
7answers
10k views

Speeding up rsync over smb

I'm backing up a Linux box over SMB to a NAS. I mount the NAS locally and then I rsync a lot of data (100GB or so). I believe it's taking an awfully long time to do it: more than 12 hours. I would ...
5
votes
10answers
16k views

Using Rsync to backup to an external drive

I'm buying an external hard drive to back up the computers in my house (finally!!). I'm hoping to use rsync. I've seen an example that does (or seems to do) exactly what I want. Something like ...
5
votes
10answers
4k views

Using rsync to quickly upload a file that is similar to another file

I'm putting together a deployment script which tars up a directory of my code, names the tar file after the current date and time, pushes that up to the server, untars it in a directory of the same ...
5
votes
3answers
865 views

rsync: can't get rid of password prompt

I'm trying to run rsync to copy files (Centos 5.2 -> Centos 5.2) and I need run using nohup it so it can run after I disconnect my ssh session. The problem is I cannot get rsync to not prompt for a ...
5
votes
6answers
2k views

Is it possible to synchronize files between servers in real-time?

I have 5 or so webservers at a loadbalancer and if a user uploads a pic it can take up to a minute to rsync it accross (crontab). This can get frustrating for some thinking it failed. Is there a way ...
5
votes
5answers
9k views

Rsync over ssh: “ERROR: module is read only” suddenly appeared

I've used from some time rsync/ssh to backup my shared host contents to my personal Synology NAS (212j for that matter), and it worked quite well. For information, I use a password-less ssh ...
5
votes
3answers
7k views

run rsync as root but keep user ownership

I would like to backup user files from one server to another with rsync. but I noticed that the user folders change to root. how can I keep the user permissions with rsync (running by root)?
5
votes
3answers
12k views

rsync via cron. How do I enable logging?

I'm backing up a remote server to another computer using rsync. In cron.daily I have a file with this: rsync -avz -e ssh root@example.com:/ /mybackup/ It uses a public / private key pair to login. ...
5
votes
3answers
7k views

Which is best for backups rsync vs rdiff vs rsnapshot

I am using rsync for backups from remote FTP to local computer. I read on internet that rsnapshot is better. Just want to know which is used in production environment
5
votes
3answers
2k views

rsync 'cannot delete non-empty directory' errors, even with --force option

When running this command: $ sudo rsync -r --delete --force --checksum --exclude=uploads /data/prep/* /data/app/ I'm getting the following output: cannot delete non-empty directory: ...
5
votes
4answers
2k views

Handling renamed files or directories in rsync

I am working on an rsync script for directory replication. I have it syncing only new and modified files or directories but am not liking the fact that its copying renamed files or directories as a ...
5
votes
4answers
5k views

rsync to compressed archive of file

I take backups of a server with rsync, but those backups are quite large, and I'd like to compress them. Is there a way to have some sort of wrapper around rsync to gunzip the file in the backup, ...
5
votes
2answers
77 views

full or incremental backup of large number of files

I have a large amount of files in both quantity and total file size. (we're talking a few terabytes). I would like to sync these files/folders once to an external backup system, then run a daily task ...