Rsync is a fast and extraordinarily versatile file copying tool. It can copy locally, to/from another host over any remote shell, or to/from a remote rsync daemon

learn more… | top users | synonyms

5
votes
3answers
5k views

Trying to setup chroot'd rsync

I'm trying to set up a backup server. I want to chroot each user (client) to its home directory, and only allow it to use sftp and rsync. I quickly discovered that I was not the only one trying to do ...
5
votes
3answers
13k views

rsync via cron. How do I enable logging?

I'm backing up a remote server to another computer using rsync. In cron.daily I have a file with this: rsync -avz -e ssh root@example.com:/ /mybackup/ It uses a public / private key pair to login. ...
5
votes
3answers
3k views

rsync 'cannot delete non-empty directory' errors, even with --force option

When running this command: $ sudo rsync -r --delete --force --checksum --exclude=uploads /data/prep/* /data/app/ I'm getting the following output: cannot delete non-empty directory: ...
5
votes
1answer
702 views

Why does rsync have to be installed on both sides to work?

This might be a silly question, but I wanted to keep some of my servers clean and prefer not to install any additional software. But I always thought rsync would copy data over SSH and would not ...
5
votes
2answers
133 views

full or incremental backup of large number of files

I have a large amount of files in both quantity and total file size. (we're talking a few terabytes). I would like to sync these files/folders once to an external backup system, then run a daily task ...
5
votes
1answer
9k views

Optimizing rsync for transferring huge files quickly

This is what I need: Rsync daemon running on source server. Destination server does a rsync pull as below: rsync -ptv rsync://source_ip:document/source_path/*.abc destination_path/ Huge binary files ...
5
votes
3answers
2k views

How to copy large (> 1 million) number of small files between two servers

I need to migrate about 1TB of data comprised of smaller files (most under 100KB) to another server. I've not even completely enumerated the files but estimates are between 1-2 million. The ...
5
votes
3answers
4k views

rsync and include / exclude. How hard can it be?

I'm trying to recursively copy a directory / file structure from one directory to another, keeping only html files. Should be a simple case of include / exclude shouldn't it? I just want to print out ...
5
votes
3answers
3k views

Rsync over ssh with root access on both sides

I have one older ubuntu server, and one newer debian server and I am migrating data from the old one to the new one. I want to use rsync to transfer data across to make final migration easier and ...
5
votes
2answers
1k views

Equivalent of `nice` for filesystem IO?

When I'm running an rsync backup job which involves copying over large files, the machine running the backup (both Mac desktops and Linux servers) grinds to a halt and the load average goes through ...
5
votes
3answers
182 views

how to make rsync not uncompressing files?

I'm using rsync to copy files between servers, I'm using -z option what compresses data only for the network transfer stage. But I would like to leave compressed files on remote machine and I have ...
5
votes
1answer
833 views

does rsync's delta algorithm work on a samba share?

I have a cron job to use rsync to sync the data on a windows samba share (that i mount using /etc/fstab on boot). My problem is that the transfer seems to take almost as long as the initial transfer ...
5
votes
2answers
838 views

Copy directory tree without empty directories?

I have the following tree # upper letters = directory # lower letters = files A |-- B |-- C |-- D |-- e <= file |-- F |-- G I need to copy this tree to another destination, ...
5
votes
1answer
1k views

bind interface for upload : scp works, not rsync

I need to upload file on a specific interface. This cannot be set-up via iptables, because my script output on several interfaces. this is working fine : scp -oBindAddress=192.168.100.1 ... but ...
5
votes
1answer
706 views

Script launching 3 copies of rsync

I have a simple script that uses rsync to copy a Postgres database to a backup location for use with Point In Time Recovery. The script is run every 2 hours via a cron job for the postgres user. For ...
5
votes
2answers
943 views

Is rsync a good candidate for failover implementation (very large dataset)?

I have a large a set of data (+100 GB) which can be stored into files. Most of the files would be in the 5k-50k range (80%), then 50k - 500k (15%) and >500k (5%). The maximum expected size of a file ...
5
votes
5answers
940 views

Windows based rsync server

Is there a way to build a Windows server what can be target for other Windows file servers with rsync ?
5
votes
2answers
2k views

Malformed rsync command on ReadyNAS

I'm trying to set up rsync backups on my ReadyNAS and I'm getting the following error: ERROR: The remote path must start with a module name not a / This error is accompanied by the following ...
5
votes
7answers
2k views

fastest backup for large files

what is the fastest way to duplicate a folder contain few files with few hundred megs size and other few megs small files? (say, /var/lib/mysql) cp or tar or rsync or...??
5
votes
2answers
2k views

rsync ionice destination

To avoid rsync to starve the system and network we use ionice when starting rsync and set the --bwlimit parameter. For example: ionice -c2 -n7 rsync -aH --bwlimit=30000 /foo root@dest.com:/ This ...
5
votes
2answers
793 views

Linux: Syncronize two directories permissions, users and groups and leave file contents untouched

In my Linux box I have two directories: work files with wrong permissions older versions of the same files with the right permissions (permissions and users and groups) I need to syncronize the ...
5
votes
4answers
1k views

rsync/ssh/rsnapshot backup — remote script execution

I have the following settings in my configuration file: (/etc/rsnapshot.conf) Script/Tool: http://www.rsnapshot.org/ backup_script ssh root@host.domain.com "sh /home/user/backup_mysql.sh" ...
5
votes
1answer
314 views

Rsync + public key authentication security

I have read several articles how to automatically backup files with Rsync and public key authentication. All of them are very similar. I just finished setting up everything and everything works fine ...
5
votes
2answers
239 views

scp and rsync do not work (ssh works)

In my Debian Squeeze server, scp and rsync do not seem to copy over anything. Works fine with another server though (so not a client problem). When using scp, it just creates an empty file and waits. ...
5
votes
3answers
2k views

lsync, unison or some other inotify auto-syncing tool..?

I have an app farm which generates thumbnails and makes them available to a web server farm using nfs. However the performance is so poor so I am going to make a local copy of the files on each web ...
5
votes
1answer
613 views

Forward-sync to HDFS? (OR continue an incomplete hdfs upload?)

Anyone have a good suggestion for doing a forward sync to HDFS? ("forward-sync" in contrast to "bi-directional sync") Basically I have a large number of files I want to put into the HDFS. Its so ...
4
votes
6answers
622 views

Is Rsync like subversion, but for a server?

I'm trying to learn how to use rsync. I want to create daily backs up of my production server. Right now I run the command rsync -azr /var/www/* www-data@backup.server.com:/var/www Now let's say ...
4
votes
5answers
136 views

send files to dozens of servers quickly

We have a cluster of machines (around 50 and growing). Each machine has a search index that needs to be updated multiple times per day. We currently update the index on each machine individually, but ...
4
votes
6answers
6k views

rsync take too long to run

I have a load balancer setup involve 2 server. these 2 server mirror each other. the main usage of the blanacer is serving static files. Let's call them Server A and Server B. Server A will retrieve ...
4
votes
1answer
4k views

Does rsync delete local files if they were deleted on source folder?

Here's what I'm trying to do: In essence, get files from one net storage device synced to another net storage device. The solution was to rsync from the source net storage to a local machine. Then ...
4
votes
2answers
1k views

rsync with --hard-links freezes

I have a large directory called servers, which contains many hard-links made by rsnapshot. That means that the structure is more or less like: ./servers ./servers/daily.0 ./servers/daily.0/file1 ...
4
votes
3answers
680 views

Backup with bash and rsync

Is there a way to auto-rename an existing file on the receiver? For example: if filename already exists, it auto-rename filename to something like filename_001, filename_002 and so on.... So far all ...
4
votes
2answers
5k views

rsync : transferring files in one direction only

Is it possible to use rsync to copy files in one direction only? For example, suppose we have: left/a.txt right/a.txt where the files are initially identical. If one then modifies right/a.txt, ...
4
votes
3answers
4k views

Escaping spaces in a remote path when using rsync over a remote SSH connection

When using SSH to connect rsync to a remote server, how do you escape spaces and such in the remote path? A simple backslash escapes the space for the local bash prompt, but on the remote machine the ...
4
votes
2answers
6k views

Speed up rsync by running multiple instances at once?

I have to transfer a big directory to another server. My problem is that i cannot use the full bandwidth that would be available, because one concurrent tcp stream does not get as fast. Basically ...
4
votes
2answers
1k views

What is the default remote shell for rsync?

Just curious, what is the default remote shell used by rsync when you're not using it in daemon mode? According to the man page it can be run via rsh or ssh. However nowhere in the man page does it ...
4
votes
3answers
2k views

Using rsync to maintain a copy of a directory with a name that changes

I am using rsync on a linux system to synchronize a directory between the local disk and an attached USB drive. The problem I am experiencing is that the 3rd party system that creates the backup ...
4
votes
1answer
782 views

rsync copies unmodified files between different file systems

I try to synchronize files between two machines using rsync, but it seems to copy all the files regardless file modification time. Two consecutive call of the command produces the same results, so it ...
4
votes
2answers
107 views

executing a command again once it stops (rsync, FreeBSD)

I started doing backups via rsync to a remote server. This works all fine and good but 40 gig take their time and as I get a dynamic IP it gets interrupted every day. So every morning when I wake up I ...
4
votes
3answers
1k views

Forcing rsync to convert file names to lower case

We are using rsync to transfer some (millions) files from a Windows (NTFS/CYGWIN) server to a Linux (RHEL) server. We would like to force all file and directory names on the Linux box to be lower ...
4
votes
1answer
898 views

Using rsync to only delete extraneous files

What's the best way of comparing two directory structures and deleting extraneous files and directories in the target location? I have a small web photo gallery app that I'm developing. Users add ...
4
votes
2answers
6k views

How can I prevent vanishing error when using rsync with an nfs mount?

I'm in the process of moving files from an Ubuntu server to a Snow Leopard server. The Ubuntu server has an NFS share of around 6TB which I'd like to clone to the Snow Leopard server. I mounted the ...
4
votes
3answers
2k views

Chunking large rsync transfers?

We use rsync to update a mirror of our primary file server to an off-site colocated backup server. One of the issues we currently have is that our file server has > 1TB of mostly smaller files (in ...
4
votes
1answer
2k views

Using a shell script to check if rsync made any changes

I've got a bash script that rsync's two directories. Sometimes there's a change, odds are there's nothing. I want to run a command only if rsync actually made a change (add/update a file). ...
4
votes
2answers
6k views

Freenas — Choosing the right redundant storage regime (RAID, ZFS, Rsync?)

I've been running a freenas box for a few months as a proof of concept and all seems to be well now reliability wise. So it's time to up the storage capacity and build it into production mode. I was ...
4
votes
3answers
293 views

rsync many small files with long filenames takes a lot of bandwidth

I have a file storage server which stores files on disk using the file's sha256 hash as the filename, along with the file extension, and in three levels of directories, e.g. a PDF file with sha256 ...
4
votes
1answer
165 views

Preserving existing destination symlinks with rsync

I am using rsync to keep a local debian repository updated. Recently, the disk I am using to store it started running low on space, so I decided to use symlinks to allow me to move some of the ...
4
votes
2answers
5k views

How to fix corrupt packet error for with rsync for (relatively) large files?

Trying to update files on a server, with the rsync command: rsync -ravq -e "ssh -o ConnectTimeout=2 -o ServerAliveInterval=2 -ServerAliveCountMax=2" --delete ./local_dir user@$SERVER:/dest_dir ...
4
votes
1answer
1k views

Why does rsync forks itself? And why one such forked process is almost kinda idle (as seen in iotop)?

This is referring to the matter stated here and I am also experiencing the same. In one of my servers, I ran an rsync, to backup a huge directory (size greater than 300 Gb) to a different disk, ...
4
votes
3answers
5k views

Using rsync from msysgit for binary files

I have three windows boxes that are connected to a linux box on my network in various ways (one's local to the network, one is a laptop and may or may not be local, and the other is my work desktop). ...