Rsync is a fast and extraordinarily versatile file copying tool. It can copy locally, to/from another host over any remote shell, or to/from a remote rsync daemon

learn more… | top users | synonyms

7
votes
4answers
2k views

How can I easily confirm in Linux that two separate directories have the exact same contents?

CentOS 5.x Mq question seemed similar to this one but I wasn't sure... I have two servers (completely isolated from each other), each with a directory and sub-directories that should have the same ...
7
votes
9answers
2k views

Copy 10 million images in a single folder to another server

Now I know you shouldn't ever put 10 million files into a single directory to begin with. Blame it on the developers, but as it stands that’s where I am at. We will be fixing it and moving them into ...
7
votes
6answers
6k views

improving rsync backup performance

What are the best techniques to improve rsync over ssh mirroring between unix boxes, assuming that one system will always have the master copy and the other system will always have a recent copy (less ...
7
votes
3answers
10k views

Which is best for backups rsync vs rdiff vs rsnapshot

I am using rsync for backups from remote FTP to local computer. I read on internet that rsnapshot is better. Just want to know which is used in production environment
7
votes
5answers
4k views

Forcing rsync to non-interactive mode

I would like to use rsync within a python script. I'm calling it using the subprocess module, and authenticating using public keys stored at the authorized_key file on the remote machine. The only ...
7
votes
2answers
2k views

Using rsync to only delete extraneous files

What's the best way of comparing two directory structures and deleting extraneous files and directories in the target location? I have a small web photo gallery app that I'm developing. Users add &...
7
votes
3answers
5k views

Rsync over ssh with root access on both sides

I have one older ubuntu server, and one newer debian server and I am migrating data from the old one to the new one. I want to use rsync to transfer data across to make final migration easier and ...
7
votes
3answers
2k views

Tell Rsync to Skip Current file During Transfer?

Is there a way to tell rsync to skip its current file while a sync is in progress, maybe by sending it a particular signal? I already know about ignoring based on patterns, but this would be handy to ...
7
votes
1answer
2k views

Why does rsync forks itself? And why one such forked process is almost kinda idle (as seen in iotop)?

This is referring to the matter stated here and I am also experiencing the same. In one of my servers, I ran an rsync, to backup a huge directory (size greater than 300 Gb) to a different disk, ...
7
votes
2answers
3k views

rsync ionice destination

To avoid rsync to starve the system and network we use ionice when starting rsync and set the --bwlimit parameter. For example: ionice -c2 -n7 rsync -aH --bwlimit=30000 /foo root@dest.com:/ This ...
7
votes
3answers
9k views

Using rsync from msysgit for binary files

I have three windows boxes that are connected to a linux box on my network in various ways (one's local to the network, one is a laptop and may or may not be local, and the other is my work desktop). ...
7
votes
1answer
2k views

rsync xattr oddness between mac & linux?

Hullo all - I'm using rsync to back up from my mac to a fileserver runnning linux, and I see the following error: rsync -e ssh -vaxE --delete --ignore-errors 192.168.1.3:/bkup/mac/Users/dave/ /Users/...
6
votes
4answers
2k views

How to copy a 200GB file faster?

I got a 200GB .tgz file on server A(RHEL 5.2). I wanna transfer that file to server B (RHEL 5.3). Server B is on ESXi 4 Update1. I gave 10GB to that Server B VM, with 4 vCPUs. Both Server A and ...
6
votes
10answers
4k views

Using rsync to quickly upload a file that is similar to another file

I'm putting together a deployment script which tars up a directory of my code, names the tar file after the current date and time, pushes that up to the server, untars it in a directory of the same ...
6
votes
6answers
8k views

Rsync Remote to Remote

I'm looking for a solution for using rsync between 2 remote servers. It seems like its not possible. Does any one know why its not possible? I ask that because I think if I know the reason, maybe I ...
6
votes
3answers
7k views

Trying to setup chroot'd rsync

I'm trying to set up a backup server. I want to chroot each user (client) to its home directory, and only allow it to use sftp and rsync. I quickly discovered that I was not the only one trying to do ...
6
votes
2answers
2k views

rsync with --hard-links freezes

I have a large directory called servers, which contains many hard-links made by rsnapshot. That means that the structure is more or less like: ./servers ./servers/daily.0 ./servers/daily.0/file1 ./...
6
votes
2answers
5k views

rsync: How to exclude Dotfiles only in topmost directory?

When performing backups under Bash with rsync, I'm trying to exclude all dotfiles and hidden directories in the top directory, but not those in otherwise targeted directories. For example: /copyme.c ...
6
votes
1answer
1k views

Why does rsync have to be installed on both sides to work?

This might be a silly question, but I wanted to keep some of my servers clean and prefer not to install any additional software. But I always thought rsync would copy data over SSH and would not ...
6
votes
4answers
270 views

rsync directory so all changes appear atomically

I do some nightly and weekly mirrors of frequently used repositories for the local network. On a few occasions, someone has tried to do an update while the rsync is happening and failed because the ...
6
votes
2answers
2k views

What is the default remote shell for rsync?

Just curious, what is the default remote shell used by rsync when you're not using it in daemon mode? According to the man page it can be run via rsh or ssh. However nowhere in the man page does it ...
6
votes
3answers
12k views

How can I prevent vanishing error when using rsync with an nfs mount?

I'm in the process of moving files from an Ubuntu server to a Snow Leopard server. The Ubuntu server has an NFS share of around 6TB which I'd like to clone to the Snow Leopard server. I mounted the ...
6
votes
1answer
11k views

Does rsync overwrite the new file

i use rsync -avh I just want to know that if the file on destination already exists does the rsync overwrite the new file or not
6
votes
4answers
2k views

rsync - report only uploaded files

I have a problem with rsync: either it shows me all files in the directories I am rsyncing (if using the flag -v) or nothing at all (when not using -v). I would like to have rsync quiet except for ...
6
votes
3answers
8k views

Rsync fails to backup some files, giving error 22

I'm trying to backup my whole server using a simple rsync on a NFS mounted volume on '/nas' Here's my rsync command rsync -sav -S --stats -H --numeric-ids --delete -D --exclude-from="/usr/local/...
6
votes
1answer
3k views

Using a shell script to check if rsync made any changes

I've got a bash script that rsync's two directories. Sometimes there's a change, odds are there's nothing. I want to run a command only if rsync actually made a change (add/update a file). ...
6
votes
6answers
2k views

Remote file copy util (like rsync) but that will take account of data already copied (in this session)?

Let's say I have a directory with 2 files, both are identical and quite large (e.g. 2GB ea.) I want to rsync that directory to a remote host. As I understand it (and I could be wrong), rsync ...
6
votes
2answers
554 views

rsync for backup considered dangerous?

In our research group, we have the need to backup data acquired on the MRI scanner in a way that will preserve any scan ever acquired (even though data might be deleted from the scanner due to space ...
6
votes
2answers
3k views

Cron and rsync using file lock but is there a better way on Mac OS X Server?

I have this bash script and is run via cron on a regular interval: #!/bin/bash RSYNC=/usr/bin/rsync SSH=/usr/bin/ssh KEY=/Users/admin/Documents/Backup/rsync-key RUSER=philosophy RHOST=example.com ...
6
votes
7answers
4k views

rsyncing a bootable backup

jwz mentions PSA: Backups a backup system that's fairly clever. Your backup method is to essentially restore to the replacement drive. This seems to work simply on OSX. What all do I need to do to ...
5
votes
7answers
14k views

Speeding up rsync over smb

I'm backing up a Linux box over SMB to a NAS. I mount the NAS locally and then I rsync a lot of data (100GB or so). I believe it's taking an awfully long time to do it: more than 12 hours. I would ...
5
votes
3answers
1k views

rsync: can't get rid of password prompt

I'm trying to run rsync to copy files (Centos 5.2 -> Centos 5.2) and I need run using nohup it so it can run after I disconnect my ssh session. The problem is I cannot get rsync to not prompt for a ...
5
votes
6answers
2k views

Is it possible to synchronize files between servers in real-time?

I have 5 or so webservers at a loadbalancer and if a user uploads a pic it can take up to a minute to rsync it accross (crontab). This can get frustrating for some thinking it failed. Is there a way ...
5
votes
2answers
12k views

rsync “Operation not permitted”

I am trying to so a simple rsync to a USB hard disc, the command is rsync -az --modify-window=2 /home /mnt/exthd/tmp/ I am running this as root. but I get a lot of errors that all seem to relate ...
5
votes
1answer
10k views

Force rsync to copy symlink, even if referent does not exist

I am getting this message with rsync: symlink has no referent: "/a/path" Well, rsync is right: in the source filesystem, the (symbolic) link is pointing to a non-existent location. That is intended....
5
votes
6answers
8k views

rsync take too long to run

I have a load balancer setup involve 2 server. these 2 server mirror each other. the main usage of the blanacer is serving static files. Let's call them Server A and Server B. Server A will retrieve ...
5
votes
3answers
10k views

How to set file/folder permissions using Rsync from Windows to Linux

I want to transfer lots of files/folders from Windows to Linux using Rsync. On linux server(destination), I want the file permission set to 644, and folder permission set to 755. If possible, I want ...
5
votes
2answers
396 views

full or incremental backup of large number of files

I have a large amount of files in both quantity and total file size. (we're talking a few terabytes). I would like to sync these files/folders once to an external backup system, then run a daily task ...
5
votes
3answers
2k views

How to copy large (> 1 million) number of small files between two servers

I need to migrate about 1TB of data comprised of smaller files (most under 100KB) to another server. I've not even completely enumerated the files but estimates are between 1-2 million. The ...
5
votes
3answers
8k views

Escaping spaces in a remote path when using rsync over a remote SSH connection

When using SSH to connect rsync to a remote server, how do you escape spaces and such in the remote path? A simple backslash escapes the space for the local bash prompt, but on the remote machine the ...
5
votes
3answers
4k views

rsync and include / exclude. How hard can it be?

I'm trying to recursively copy a directory / file structure from one directory to another, keeping only html files. Should be a simple case of include / exclude shouldn't it? I just want to print out ...
5
votes
2answers
1k views

Equivalent of `nice` for filesystem IO?

When I'm running an rsync backup job which involves copying over large files, the machine running the backup (both Mac desktops and Linux servers) grinds to a halt and the load average goes through ...
5
votes
3answers
244 views

how to make rsync not uncompressing files?

I'm using rsync to copy files between servers, I'm using -z option what compresses data only for the network transfer stage. But I would like to leave compressed files on remote machine and I have ...
5
votes
1answer
1k views

does rsync's delta algorithm work on a samba share?

I have a cron job to use rsync to sync the data on a windows samba share (that i mount using /etc/fstab on boot). My problem is that the transfer seems to take almost as long as the initial transfer ...
5
votes
3answers
12k views

Complex includes/excludes with rsync

I'm trying to work out the rsync filter syntax to perform complex include/excludes, and trying to achieve the following: Include / Exclude /home Include /home/user1/* Include /home/user2/subdir/* I'...
5
votes
1answer
3k views

bind interface for upload : scp works, not rsync

I need to upload file on a specific interface. This cannot be set-up via iptables, because my script output on several interfaces. this is working fine : scp -oBindAddress=192.168.100.1 ... but ...
5
votes
1answer
916 views

Script launching 3 copies of rsync

I have a simple script that uses rsync to copy a Postgres database to a backup location for use with Point In Time Recovery. The script is run every 2 hours via a cron job for the postgres user. For ...
5
votes
2answers
1k views

Is rsync a good candidate for failover implementation (very large dataset)?

I have a large a set of data (+100 GB) which can be stored into files. Most of the files would be in the 5k-50k range (80%), then 50k - 500k (15%) and >500k (5%). The maximum expected size of a file ...
5
votes
5answers
1k views

Windows based rsync server

Is there a way to build a Windows server what can be target for other Windows file servers with rsync ?
5
votes
2answers
18k views

rsync only files created or modified after a date and time

The goal is to rsync (or using any other tools that give the desired result) only files that are created or modified after a date. It should work with filenames containing spaces (and/or other ...