Rsync is a fast and extraordinarily versatile file copying tool. It can copy locally, to/from another host over any remote shell, or to/from a remote rsync daemon

learn more… | top users | synonyms

5
votes
3answers
5k views

Trying to setup chroot'd rsync

I'm trying to set up a backup server. I want to chroot each user (client) to its home directory, and only allow it to use sftp and rsync. I quickly discovered that I was not the only one trying to do ...
5
votes
3answers
13k views

rsync via cron. How do I enable logging?

I'm backing up a remote server to another computer using rsync. In cron.daily I have a file with this: rsync -avz -e ssh root@example.com:/ /mybackup/ It uses a public / private key pair to login. ...
5
votes
3answers
3k views

rsync 'cannot delete non-empty directory' errors, even with --force option

When running this command: $ sudo rsync -r --delete --force --checksum --exclude=uploads /data/prep/* /data/app/ I'm getting the following output: cannot delete non-empty directory: ...
5
votes
1answer
10k views

Optimizing rsync for transferring huge files quickly

This is what I need: Rsync daemon running on source server. Destination server does a rsync pull as below: rsync -ptv rsync://source_ip:document/source_path/*.abc destination_path/ Huge binary files ...
5
votes
1answer
724 views

Why does rsync have to be installed on both sides to work?

This might be a silly question, but I wanted to keep some of my servers clean and prefer not to install any additional software. But I always thought rsync would copy data over SSH and would not ...
5
votes
2answers
149 views

full or incremental backup of large number of files

I have a large amount of files in both quantity and total file size. (we're talking a few terabytes). I would like to sync these files/folders once to an external backup system, then run a daily task ...
5
votes
3answers
2k views

How to copy large (> 1 million) number of small files between two servers

I need to migrate about 1TB of data comprised of smaller files (most under 100KB) to another server. I've not even completely enumerated the files but estimates are between 1-2 million. The ...
5
votes
3answers
4k views

rsync and include / exclude. How hard can it be?

I'm trying to recursively copy a directory / file structure from one directory to another, keeping only html files. Should be a simple case of include / exclude shouldn't it? I just want to print out ...
5
votes
3answers
3k views

Rsync over ssh with root access on both sides

I have one older ubuntu server, and one newer debian server and I am migrating data from the old one to the new one. I want to use rsync to transfer data across to make final migration easier and ...
5
votes
2answers
1k views

Equivalent of `nice` for filesystem IO?

When I'm running an rsync backup job which involves copying over large files, the machine running the backup (both Mac desktops and Linux servers) grinds to a halt and the load average goes through ...
5
votes
3answers
183 views

how to make rsync not uncompressing files?

I'm using rsync to copy files between servers, I'm using -z option what compresses data only for the network transfer stage. But I would like to leave compressed files on remote machine and I have ...
5
votes
1answer
881 views

does rsync's delta algorithm work on a samba share?

I have a cron job to use rsync to sync the data on a windows samba share (that i mount using /etc/fstab on boot). My problem is that the transfer seems to take almost as long as the initial transfer ...
5
votes
2answers
983 views

Copy directory tree without empty directories?

I have the following tree # upper letters = directory # lower letters = files A |-- B |-- C |-- D |-- e <= file |-- F |-- G I need to copy this tree to another destination, ...
5
votes
1answer
1k views

bind interface for upload : scp works, not rsync

I need to upload file on a specific interface. This cannot be set-up via iptables, because my script output on several interfaces. this is working fine : scp -oBindAddress=192.168.100.1 ... but ...
5
votes
1answer
717 views

Script launching 3 copies of rsync

I have a simple script that uses rsync to copy a Postgres database to a backup location for use with Point In Time Recovery. The script is run every 2 hours via a cron job for the postgres user. For ...
5
votes
2answers
953 views

Is rsync a good candidate for failover implementation (very large dataset)?

I have a large a set of data (+100 GB) which can be stored into files. Most of the files would be in the 5k-50k range (80%), then 50k - 500k (15%) and >500k (5%). The maximum expected size of a file ...
5
votes
5answers
951 views

Windows based rsync server

Is there a way to build a Windows server what can be target for other Windows file servers with rsync ?
5
votes
2answers
2k views

Malformed rsync command on ReadyNAS

I'm trying to set up rsync backups on my ReadyNAS and I'm getting the following error: ERROR: The remote path must start with a module name not a / This error is accompanied by the following ...
5
votes
7answers
2k views

fastest backup for large files

what is the fastest way to duplicate a folder contain few files with few hundred megs size and other few megs small files? (say, /var/lib/mysql) cp or tar or rsync or...??
5
votes
2answers
2k views

rsync ionice destination

To avoid rsync to starve the system and network we use ionice when starting rsync and set the --bwlimit parameter. For example: ionice -c2 -n7 rsync -aH --bwlimit=30000 /foo root@dest.com:/ This ...
5
votes
1answer
7k views

What exactly will --delete-excluded do for rsync?

I use rsync with great pleasure for backing up my servers, although I do have a question about the --delete-exclude parameter. /usr/bin/rsync -a --delete --numeric-ids --relative --delete-excluded ...
5
votes
2answers
1k views

Can I make rsync output summary only?

Because I use rsync to backup a directory which is very big, containing many sub-dirs and files, so I dont want to see the "incremental file list", I want to know the summary in the end only. but if I ...
5
votes
2answers
808 views

Linux: Syncronize two directories permissions, users and groups and leave file contents untouched

In my Linux box I have two directories: work files with wrong permissions older versions of the same files with the right permissions (permissions and users and groups) I need to syncronize the ...
5
votes
4answers
1k views

rsync/ssh/rsnapshot backup — remote script execution

I have the following settings in my configuration file: (/etc/rsnapshot.conf) Script/Tool: http://www.rsnapshot.org/ backup_script ssh root@host.domain.com "sh /home/user/backup_mysql.sh" ...
5
votes
1answer
368 views

Rsync + public key authentication security

I have read several articles how to automatically backup files with Rsync and public key authentication. All of them are very similar. I just finished setting up everything and everything works fine ...
5
votes
2answers
246 views

scp and rsync do not work (ssh works)

In my Debian Squeeze server, scp and rsync do not seem to copy over anything. Works fine with another server though (so not a client problem). When using scp, it just creates an empty file and waits. ...
5
votes
3answers
2k views

lsync, unison or some other inotify auto-syncing tool..?

I have an app farm which generates thumbnails and makes them available to a web server farm using nfs. However the performance is so poor so I am going to make a local copy of the files on each web ...
5
votes
1answer
626 views

Forward-sync to HDFS? (OR continue an incomplete hdfs upload?)

Anyone have a good suggestion for doing a forward sync to HDFS? ("forward-sync" in contrast to "bi-directional sync") Basically I have a large number of files I want to put into the HDFS. Its so ...
4
votes
6answers
624 views

Is Rsync like subversion, but for a server?

I'm trying to learn how to use rsync. I want to create daily backs up of my production server. Right now I run the command rsync -azr /var/www/* www-data@backup.server.com:/var/www Now let's say ...
4
votes
5answers
137 views

send files to dozens of servers quickly

We have a cluster of machines (around 50 and growing). Each machine has a search index that needs to be updated multiple times per day. We currently update the index on each machine individually, but ...
4
votes
6answers
7k views

rsync take too long to run

I have a load balancer setup involve 2 server. these 2 server mirror each other. the main usage of the blanacer is serving static files. Let's call them Server A and Server B. Server A will retrieve ...
4
votes
1answer
4k views

Does rsync delete local files if they were deleted on source folder?

Here's what I'm trying to do: In essence, get files from one net storage device synced to another net storage device. The solution was to rsync from the source net storage to a local machine. Then ...
4
votes
2answers
1k views

rsync with --hard-links freezes

I have a large directory called servers, which contains many hard-links made by rsnapshot. That means that the structure is more or less like: ./servers ./servers/daily.0 ./servers/daily.0/file1 ...
4
votes
3answers
688 views

Backup with bash and rsync

Is there a way to auto-rename an existing file on the receiver? For example: if filename already exists, it auto-rename filename to something like filename_001, filename_002 and so on.... So far all ...
4
votes
1answer
975 views

Can rsync display current average speed?

When I'm transferring large quantities of data using rsync, it would be helpful if I could have the average speed up until now at a glance, rather than a bunch of different speeds for each file.
4
votes
2answers
5k views

rsync : transferring files in one direction only

Is it possible to use rsync to copy files in one direction only? For example, suppose we have: left/a.txt right/a.txt where the files are initially identical. If one then modifies right/a.txt, ...
4
votes
3answers
5k views

Escaping spaces in a remote path when using rsync over a remote SSH connection

When using SSH to connect rsync to a remote server, how do you escape spaces and such in the remote path? A simple backslash escapes the space for the local bash prompt, but on the remote machine the ...
4
votes
2answers
3k views

Can I use rsync to create a list of only changed files?

I am using rsync in a bash script to keep files in sync between a few servers and a NAS. One issue I have run into is trying to generate a list of the files that have changed from the during the ...
4
votes
2answers
6k views

Speed up rsync by running multiple instances at once?

I have to transfer a big directory to another server. My problem is that i cannot use the full bandwidth that would be available, because one concurrent tcp stream does not get as fast. Basically ...
4
votes
2answers
1k views

What is the default remote shell for rsync?

Just curious, what is the default remote shell used by rsync when you're not using it in daemon mode? According to the man page it can be run via rsh or ssh. However nowhere in the man page does it ...
4
votes
3answers
2k views

Using rsync to maintain a copy of a directory with a name that changes

I am using rsync on a linux system to synchronize a directory between the local disk and an attached USB drive. The problem I am experiencing is that the 3rd party system that creates the backup ...
4
votes
1answer
819 views

rsync copies unmodified files between different file systems

I try to synchronize files between two machines using rsync, but it seems to copy all the files regardless file modification time. Two consecutive call of the command produces the same results, so it ...
4
votes
2answers
108 views

executing a command again once it stops (rsync, FreeBSD)

I started doing backups via rsync to a remote server. This works all fine and good but 40 gig take their time and as I get a dynamic IP it gets interrupted every day. So every morning when I wake up I ...
4
votes
3answers
1k views

Forcing rsync to convert file names to lower case

We are using rsync to transfer some (millions) files from a Windows (NTFS/CYGWIN) server to a Linux (RHEL) server. We would like to force all file and directory names on the Linux box to be lower ...
4
votes
1answer
931 views

Using rsync to only delete extraneous files

What's the best way of comparing two directory structures and deleting extraneous files and directories in the target location? I have a small web photo gallery app that I'm developing. Users add ...
4
votes
2answers
7k views

How can I prevent vanishing error when using rsync with an nfs mount?

I'm in the process of moving files from an Ubuntu server to a Snow Leopard server. The Ubuntu server has an NFS share of around 6TB which I'd like to clone to the Snow Leopard server. I mounted the ...
4
votes
3answers
3k views

Chunking large rsync transfers?

We use rsync to update a mirror of our primary file server to an off-site colocated backup server. One of the issues we currently have is that our file server has > 1TB of mostly smaller files (in ...
4
votes
1answer
2k views

Using a shell script to check if rsync made any changes

I've got a bash script that rsync's two directories. Sometimes there's a change, odds are there's nothing. I want to run a command only if rsync actually made a change (add/update a file). ...
4
votes
2answers
6k views

Freenas — Choosing the right redundant storage regime (RAID, ZFS, Rsync?)

I've been running a freenas box for a few months as a proof of concept and all seems to be well now reliability wise. So it's time to up the storage capacity and build it into production mode. I was ...
4
votes
3answers
322 views

rsync many small files with long filenames takes a lot of bandwidth

I have a file storage server which stores files on disk using the file's sha256 hash as the filename, along with the file extension, and in three levels of directories, e.g. a PDF file with sha256 ...