Rsync is a fast and extraordinarily versatile file copying tool. It can copy locally, to/from another host over any remote shell, or to/from a remote rsync daemon

learn more… | top users | synonyms

3
votes
3answers
4k views

simple rsync in crontab without password

I would like to run the following: (Centos 5.4) rsync -avz /source_folder 192.168.1.1:/destination_folder right now I'm running it from CLI but eventually I would like to run it from crontab ...
98
votes
15answers
148k views

How to use rsync over FTP

Any unix: I have the following cmd line which works fine. rsync -avr -e ssh /home/dir user@example.com:/home/ But I need to set it up now to rsync to a remote server that only has an FTP server on ...
21
votes
5answers
26k views

Is it possible to use rsync over sftp (without an ssh shell)?

Rsync over ssh, works great every time. However, trying to rsync to a host which allows only sftp logins, but not ssh logins, provides the following error: rsync -av /source ssh ...
7
votes
3answers
2k views

Why don't people just use rsync to back up vmware guests?

If I am running a modern vmware ESXi system, I can drop in a statically linked rsync binary and rsync files to any destination over SSH. I'm trying to understand why most (all ?) backup of vmware ...
3
votes
2answers
1k views

Git-style incremental backup with rsync

I'm trying to setup a backup script on Ubuntu. Every day I want to copy my local source directory to a backup directory on a remote server uniquely named with the date. (e.g., backup-jan1/, ...
14
votes
13answers
9k views

Complete restore of linux system

I am familiar with using rsync to back up various files from my system but what is the best way to completely restore a machine. What I have tried in the past is: Do a basic format/reinstall from ...
10
votes
8answers
10k views

Copy large file from one Linux server to another

I'm attempting to copy a 75 gigabyte tgz (mysql lvm snapshot) from a Linux server in our LA data center to another Linux server in our NY data center over a 10MB link. I am getting about 20-30Kb/s ...
22
votes
3answers
26k views

Why is my rsync so slow?

My Laptop and my workstation are both connected to a Gigabit Switch. Both are running Linux. But when I copy files with rsync, it performs badly. I get about 22 MB/s. Shouldn't I theoretically get ...
18
votes
9answers
19k views

Best way to compare (diff) a full directory structure?

What's the best way to compare directory structures? I have a backup utility which uses rsync. I want to tell the exact differences (in terms of file sizes and last-changed dates) between the source ...
7
votes
3answers
4k views

compare 2 directories and copy differences in a 3rd directory

Running ubuntu 12.04, I want to compare 2 directories, say folder1/ and folder2/ and copy any files that are different to folder3/. There are also nested files, so matching subdirectories should be ...
17
votes
6answers
18k views

protocol version mismatch — is your shell clean?

When following the instructions to do rsync backups given here: http://troy.jdmz.net/rsync/index.html I get the error "protocol version mismatch -- is your shell clean?" I read somewhere that I ...
10
votes
2answers
314 views

Server migration: most efficient way

I have been tasked with migrating one of our sites between servers (two different hosts). Both environments are linux. The site streams video, so the server is currently filled with media files ...
6
votes
3answers
5k views

Rsync seems incompatible with .bashrc (causes “is your shell clean?”)

It turns out rsync can't work with a remote server which has a .bashrc file? At local client i got when run rsync: protocol version mismatch -- is your shell clean? (see the rsync man page for an ...
6
votes
2answers
8k views

How to repeatedly call rsync until files are sucessfully transferred

I'm trying to syncronize files from a remote server that is not reliable, meaning the connection tends to fail "randomly" with rsync: connection unexpectedly closed Rsync is called with ...
5
votes
7answers
10k views

Speeding up rsync over smb

I'm backing up a Linux box over SMB to a NAS. I mount the NAS locally and then I rsync a lot of data (100GB or so). I believe it's taking an awfully long time to do it: more than 12 hours. I would ...
7
votes
6answers
5k views

what is the fastest and most reliable way of transferring a lot of files?

I'm trying to transfer about 100k files totaling 90gb. Right now I'm using rsync daemon but its slow 3.4mb/s and I need to do this a number of times. I'm wondering what options do I have that would ...
3
votes
3answers
6k views

Complex includes/excludes with rsync

I'm trying to work out the rsync filter syntax to perform complex include/excludes, and trying to achieve the following: Include / Exclude /home Include /home/user1/* Include /home/user2/subdir/* ...
3
votes
2answers
2k views

rsync with script supplied password

I need to have a script periodically rsync files from a remote host via ssh. The account which the script runs under does not have home directory on the remote host, so I'm not able to store my ...
2
votes
2answers
879 views

How do I reserve bandwidth for rsync?

Currently nginx uses all my bandwidth. How do I leave some for rsync?
3
votes
2answers
1k views

rsync via multiple ssh connections with a key

I need to do a rsync from a target server with a server in the middle and I have the same public key on both the target and middle server. I want to use that key so I don't have to do password ...
3
votes
1answer
761 views

Rsync: dealing with absolute symlinks

Having read through the rsync manpage, I can't see any way for rsync to automagically transform absolute symlinks between machines, i.e Server A file -> /home/username/path/file Server B file ...
2
votes
2answers
180 views

Possibility of WAN Optimization for SSH traffic

While I do understand that SSH by itself is a very low bandwidth utilizing protocol, it is sometimes choking up bandwidth during peak hours in our office environment. I want to know if at all it is ...
2
votes
7answers
2k views

Backup - rsync or tar

We're looking to backup about 100gb+ of data containing small files (10kb+) each. The backup needs to be done as fast as possible to another harddrive weekly. Which is the better (especially speed ...
1
vote
1answer
3k views

rsync error unexplained error (code 255) at io.c

I was using a script to perform rsync in sudo crontab. The script does a 2-way rsync (from serverA to serverB and reverse). After i reboot both the server machines, the rsync is not working in sudo ...
1
vote
3answers
1k views

How to make an ISO copy of Linux-filesystem and user files of VPS Debian based?

I have a Debian-Based VPS on some hosting. I want to migrate from it and i need to make a full copy of all Linux-filesystem (and installed packages) + all home directory with website files. And then ...
0
votes
6answers
1k views

realtime sync of files between instances

In my scenario, i have 2 ec2 instances and need to sync the files between them in realtime. Can I do this with rsync? Can anyone please suggest how to schedule a rsync?
0
votes
2answers
83 views

Need help w/ rsync

I need some help getting my rync command syntax working. I need to clone the /home/user/public_html folder on a remote server to the server I am logged into via SSH, copying the folder into the same ...
78
votes
15answers
86k views

Copying a large directory tree locally? cp or rsync?

I have to copy a large directory tree, about 1.8 TB. It's all local. Out of habit I'd use rsync, however I wonder if there's much point, and if I should rather use cp. I'm worried about permissions ...
46
votes
15answers
42k views

Showing total progress in rsync: is it possible?

I have searched for this option already, but have only found solutions that involve custom patching. The fact that it does not show in --help and no more info can be found probably indicates the ...
44
votes
3answers
31k views

What is archive mode in rsync?

I know you can use -a or --archive to activate archive mode when using rsync. Unfortunately, I have no idea what archive mode is supposed to do, and the man page is not at all explicit about what this ...
42
votes
18answers
40k views

Why hasn't rsync caught on in the Windows world? [closed]

It seems that rsync is the de-facto standard for efficient file backup and sync in Unix/Linux. Does anyone have any thoughts on why it wouldn't have caught on in the Windows world? Why hasn't it ...
13
votes
9answers
18k views

Sync LVM snapshots to backup server

I have a number of Xen virtual machines running on a number of Linux servers. These VMs store their disk images in Linux LVM volumes with device names along the lines of /dev/xenVG/SERVER001OS and so ...
15
votes
9answers
5k views

Tool or script to detect moved or renamed files on Linux prior to a backup

Basically I am searching to see if there exists a tool or script that can detect moved or renamed files so that I can get a list of renamed/moved files and apply the same operation on the other end of ...
6
votes
4answers
3k views

Converting UTF-8 NFD filenames to UTF-8 NFC, in either rsync or afpd

I have a home file server running FreeNAS 8. A few days ago I used rsync to upload my entire iTunes library from Mac so that I could load my library over the network instead of off a slow USB drive. ...
3
votes
1answer
1k views

Rsnapshot to multiple destinations (or maintain hard links structure)

I'm doing a backup of multiple external servers on a local machine using rsnapshot and it's working beautifully. Now, I'd like to clone the destination snapshots on an external disk that will be ...
1
vote
4answers
2k views

Run an rsync replication script under a user other than root

How can I run an rsync backup/replication script in a user other than root while preserving permissions? This is a multi-user fileserver, where each user has a *nix account for permissions. Running ...
7
votes
4answers
11k views

rsync is just hanging, building file list

[root@centos /]# rsync -av --exclude thumbs /storage root@xx.27.1.xx:/storage root@xx.27.1.xx's password: building file list ... been sitting for about an hour... it's 135GB of images and folders ...
5
votes
3answers
2k views

lsync, unison or some other inotify auto-syncing tool..?

I have an app farm which generates thumbnails and makes them available to a web server farm using nfs. However the performance is so poor so I am going to make a local copy of the files on each web ...
5
votes
6answers
2k views

Is it possible to synchronize files between servers in real-time?

I have 5 or so webservers at a loadbalancer and if a user uploads a pic it can take up to a minute to rsync it accross (crontab). This can get frustrating for some thinking it failed. Is there a way ...
5
votes
10answers
16k views

Using Rsync to backup to an external drive

I'm buying an external hard drive to back up the computers in my house (finally!!). I'm hoping to use rsync. I've seen an example that does (or seems to do) exactly what I want. Something like ...
4
votes
2answers
2k views

sshd_config ForceCommand /usr/bin/rsync error “connection unexpectedly closed”

How should ForceCommand be configured in order to allow the backup user to execute any rsync command? With ForceCommand set to /usr/bin/rsync I receive the error below: connection unexpectedly ...
4
votes
3answers
5k views

Trying to setup chroot'd rsync

I'm trying to set up a backup server. I want to chroot each user (client) to its home directory, and only allow it to use sftp and rsync. I quickly discovered that I was not the only one trying to do ...
3
votes
7answers
7k views

Copying huge files between two remote machines - Efficiently

I have a shell script which keeps on copying huge files (2 GB to 5 GB) between remote systems. Key based authentication is used with agent-forwarding and everything works. For ex: Say the shell script ...
2
votes
1answer
1k views

Rsync over SSHFS hangs

I have sshfs setup to connect to another system (Rackspace Cloud Sites) that I don't have ssh access to (but for some reason I can use sshfs? Go figure). I'm attempting to rsync files from the sshfs ...
2
votes
1answer
5k views

rsync and --fake-super - how to preserve attributes?

I have been trying to sort out my backups, so that when files are restored the owners/groups and any other metadata are correctly restored. For my testing I have been backing up from a Ubuntu 9.10 ...
1
vote
2answers
167 views

Using rsync and which directories should I place in my exclude file when using rsync?

I would like to use a CentOS VM for a mirrored full backup of my CentOS mail server. Both are the same version and are up-to-date. I have found that without excluding certain directories the VM backup ...
9
votes
5answers
6k views

How can I use rsync with a FAT file system?

I would like to write a simple backup script that saves some data to a FAT drive. Should I reformat the drive and use a better file system or is it possible to use rsync with FAT? If so, what problems ...
4
votes
2answers
587 views

rsync not removing single quotes around array of strings --filter list

Trying to safely retrieve files from heavily infected Windows XP and 2003 machines that came to us from an acquisition. I have a similar need for automating a script that would run on PXE booted ...
4
votes
2answers
1k views

rsync with --hard-links freezes

I have a large directory called servers, which contains many hard-links made by rsnapshot. That means that the structure is more or less like: ./servers ./servers/daily.0 ./servers/daily.0/file1 ...
4
votes
3answers
2k views

rsync and --remove-source-files

I use rsync v. 3.0.4 and when I need to move something I use it with the --remove-source-files. I prefer rsync than mv. Unfortunately, when I use --remove-source-files, the directories are left on ...