Rsync is a fast and extraordinarily versatile file copying tool. It can copy locally, to/from another host over any remote shell, or to/from a remote rsync daemon

learn more… | top users | synonyms

6
votes
8answers
4k views

rsyncing a bootable backup

jwz mentions PSA: Backups a backup system that's fairly clever. Your backup method is to essentially restore to the replacement drive. This seems to work simply on OSX. What all do I need to do to ...
6
votes
2answers
2k views

Copy directory tree without empty directories?

I have the following tree # upper letters = directory # lower letters = files A |-- B |-- C |-- D |-- e <= file |-- F |-- G I need to copy this tree to another destination, ...
6
votes
6answers
2k views

Remote file copy util (like rsync) but that will take account of data already copied (in this session)?

Let's say I have a directory with 2 files, both are identical and quite large (e.g. 2GB ea.) I want to rsync that directory to a remote host. As I understand it (and I could be wrong), rsync ...
6
votes
3answers
8k views

How to fix corrupt packet error for with rsync for (relatively) large files?

Trying to update files on a server, with the rsync command: rsync -ravq -e "ssh -o ConnectTimeout=2 -o ServerAliveInterval=2 -ServerAliveCountMax=2" --delete ./local_dir user@$SERVER:/dest_dir ...
6
votes
2answers
468 views

rsync for backup considered dangerous?

In our research group, we have the need to backup data acquired on the MRI scanner in a way that will preserve any scan ever acquired (even though data might be deleted from the scanner due to space ...
6
votes
2answers
3k views

Cron and rsync using file lock but is there a better way on Mac OS X Server?

I have this bash script and is run via cron on a regular interval: #!/bin/bash RSYNC=/usr/bin/rsync SSH=/usr/bin/ssh KEY=/Users/admin/Documents/Backup/rsync-key RUSER=philosophy RHOST=example.com ...
5
votes
7answers
12k views

Speeding up rsync over smb

I'm backing up a Linux box over SMB to a NAS. I mount the NAS locally and then I rsync a lot of data (100GB or so). I believe it's taking an awfully long time to do it: more than 12 hours. I would ...
5
votes
10answers
4k views

Using rsync to quickly upload a file that is similar to another file

I'm putting together a deployment script which tars up a directory of my code, names the tar file after the current date and time, pushes that up to the server, untars it in a directory of the same ...
5
votes
3answers
962 views

rsync: can't get rid of password prompt

I'm trying to run rsync to copy files (Centos 5.2 -> Centos 5.2) and I need run using nohup it so it can run after I disconnect my ssh session. The problem is I cannot get rsync to not prompt for a ...
5
votes
6answers
2k views

Is it possible to synchronize files between servers in real-time?

I have 5 or so webservers at a loadbalancer and if a user uploads a pic it can take up to a minute to rsync it accross (crontab). This can get frustrating for some thinking it failed. Is there a way ...
5
votes
6answers
12k views

Rsync over ssh: “ERROR: module is read only” suddenly appeared

I've used from some time rsync/ssh to backup my shared host contents to my personal Synology NAS (212j for that matter), and it worked quite well. For information, I use a password-less ssh ...
5
votes
5answers
5k views

Rsync Remote to Remote

I'm looking for a solution for using rsync between 2 remote servers. It seems like its not possible. Does any one know why its not possible? I ask that because I think if I know the reason, maybe I ...
5
votes
3answers
6k views

Trying to setup chroot'd rsync

I'm trying to set up a backup server. I want to chroot each user (client) to its home directory, and only allow it to use sftp and rsync. I quickly discovered that I was not the only one trying to do ...
5
votes
6answers
7k views

rsync take too long to run

I have a load balancer setup involve 2 server. these 2 server mirror each other. the main usage of the blanacer is serving static files. Let's call them Server A and Server B. Server A will retrieve ...
5
votes
1answer
13k views

Optimizing rsync for transferring huge files quickly

This is what I need: Rsync daemon running on source server. Destination server does a rsync pull as below: rsync -ptv rsync://source_ip:document/source_path/*.abc destination_path/ Huge binary files ...
5
votes
1answer
916 views

Why does rsync have to be installed on both sides to work?

This might be a silly question, but I wanted to keep some of my servers clean and prefer not to install any additional software. But I always thought rsync would copy data over SSH and would not ...
5
votes
2answers
235 views

full or incremental backup of large number of files

I have a large amount of files in both quantity and total file size. (we're talking a few terabytes). I would like to sync these files/folders once to an external backup system, then run a daily task ...
5
votes
3answers
2k views

How to copy large (> 1 million) number of small files between two servers

I need to migrate about 1TB of data comprised of smaller files (most under 100KB) to another server. I've not even completely enumerated the files but estimates are between 1-2 million. The ...
5
votes
3answers
4k views

rsync and include / exclude. How hard can it be?

I'm trying to recursively copy a directory / file structure from one directory to another, keeping only html files. Should be a simple case of include / exclude shouldn't it? I just want to print out ...
5
votes
2answers
1k views

Equivalent of `nice` for filesystem IO?

When I'm running an rsync backup job which involves copying over large files, the machine running the backup (both Mac desktops and Linux servers) grinds to a halt and the load average goes through ...
5
votes
3answers
203 views

how to make rsync not uncompressing files?

I'm using rsync to copy files between servers, I'm using -z option what compresses data only for the network transfer stage. But I would like to leave compressed files on remote machine and I have ...
5
votes
1answer
1k views

does rsync's delta algorithm work on a samba share?

I have a cron job to use rsync to sync the data on a windows samba share (that i mount using /etc/fstab on boot). My problem is that the transfer seems to take almost as long as the initial transfer ...
5
votes
2answers
1k views

Using rsync to only delete extraneous files

What's the best way of comparing two directory structures and deleting extraneous files and directories in the target location? I have a small web photo gallery app that I'm developing. Users add ...
5
votes
1answer
2k views

Why does rsync forks itself? And why one such forked process is almost kinda idle (as seen in iotop)?

This is referring to the matter stated here and I am also experiencing the same. In one of my servers, I ran an rsync, to backup a huge directory (size greater than 300 Gb) to a different disk, ...
5
votes
1answer
2k views

bind interface for upload : scp works, not rsync

I need to upload file on a specific interface. This cannot be set-up via iptables, because my script output on several interfaces. this is working fine : scp -oBindAddress=192.168.100.1 ... but ...
5
votes
1answer
773 views

Script launching 3 copies of rsync

I have a simple script that uses rsync to copy a Postgres database to a backup location for use with Point In Time Recovery. The script is run every 2 hours via a cron job for the postgres user. For ...
5
votes
2answers
1k views

Is rsync a good candidate for failover implementation (very large dataset)?

I have a large a set of data (+100 GB) which can be stored into files. Most of the files would be in the 5k-50k range (80%), then 50k - 500k (15%) and >500k (5%). The maximum expected size of a file ...
5
votes
5answers
1k views

Windows based rsync server

Is there a way to build a Windows server what can be target for other Windows file servers with rsync ?
5
votes
2answers
3k views

Malformed rsync command on ReadyNAS

I'm trying to set up rsync backups on my ReadyNAS and I'm getting the following error: ERROR: The remote path must start with a module name not a / This error is accompanied by the following ...
5
votes
7answers
2k views

fastest backup for large files

what is the fastest way to duplicate a folder contain few files with few hundred megs size and other few megs small files? (say, /var/lib/mysql) cp or tar or rsync or...??
5
votes
2answers
2k views

rsync ionice destination

To avoid rsync to starve the system and network we use ionice when starting rsync and set the --bwlimit parameter. For example: ionice -c2 -n7 rsync -aH --bwlimit=30000 /foo root@dest.com:/ This ...
5
votes
2answers
3k views

sshd_config ForceCommand /usr/bin/rsync error “connection unexpectedly closed”

How should ForceCommand be configured in order to allow the backup user to execute any rsync command? With ForceCommand set to /usr/bin/rsync I receive the error below: connection unexpectedly ...
5
votes
3answers
6k views

Using rsync from msysgit for binary files

I have three windows boxes that are connected to a linux box on my network in various ways (one's local to the network, one is a laptop and may or may not be local, and the other is my work desktop). ...
5
votes
2answers
909 views

Linux: Syncronize two directories permissions, users and groups and leave file contents untouched

In my Linux box I have two directories: work files with wrong permissions older versions of the same files with the right permissions (permissions and users and groups) I need to syncronize the ...
5
votes
1answer
2k views

rsync xattr oddness between mac & linux?

Hullo all - I'm using rsync to back up from my mac to a fileserver runnning linux, and I see the following error: rsync -e ssh -vaxE --delete --ignore-errors 192.168.1.3:/bkup/mac/Users/dave/ ...
5
votes
4answers
2k views

rsync/ssh/rsnapshot backup — remote script execution

I have the following settings in my configuration file: (/etc/rsnapshot.conf) Script/Tool: http://www.rsnapshot.org/ backup_script ssh root@host.domain.com "sh /home/user/backup_mysql.sh" ...
5
votes
1answer
566 views

Rsync + public key authentication security

I have read several articles how to automatically backup files with Rsync and public key authentication. All of them are very similar. I just finished setting up everything and everything works fine ...
5
votes
3answers
1k views

rsync - report only uploaded files

I have a problem with rsync: either it shows me all files in the directories I am rsyncing (if using the flag -v) or nothing at all (when not using -v). I would like to have rsync quiet except for ...
5
votes
2answers
286 views

scp and rsync do not work (ssh works)

In my Debian Squeeze server, scp and rsync do not seem to copy over anything. Works fine with another server though (so not a client problem). When using scp, it just creates an empty file and waits. ...
5
votes
3answers
2k views

lsync, unison or some other inotify auto-syncing tool..?

I have an app farm which generates thumbnails and makes them available to a web server farm using nfs. However the performance is so poor so I am going to make a local copy of the files on each web ...
5
votes
1answer
726 views

Forward-sync to HDFS? (OR continue an incomplete hdfs upload?)

Anyone have a good suggestion for doing a forward sync to HDFS? ("forward-sync" in contrast to "bi-directional sync") Basically I have a large number of files I want to put into the HDFS. Its so ...
4
votes
7answers
13k views

Copying huge files between two remote machines - Efficiently

I have a shell script which keeps on copying huge files (2 GB to 5 GB) between remote systems. Key based authentication is used with agent-forwarding and everything works. For ex: Say the shell script ...
4
votes
5answers
5k views

How do I copy a directory tree but not the files in Linux?

I want to copy about 200 directories & subdirectories from one location to another but I don't want to copy the thousands of files within those directories. I am on Linux. Note: I don't have ...
4
votes
6answers
633 views

Is Rsync like subversion, but for a server?

I'm trying to learn how to use rsync. I want to create daily backs up of my production server. Right now I run the command rsync -azr /var/www/* www-data@backup.server.com:/var/www Now let's say ...
4
votes
5answers
157 views

send files to dozens of servers quickly

We have a cluster of machines (around 50 and growing). Each machine has a search index that needs to be updated multiple times per day. We currently update the index on each machine individually, but ...
4
votes
2answers
2k views

rsync with --hard-links freezes

I have a large directory called servers, which contains many hard-links made by rsnapshot. That means that the structure is more or less like: ./servers ./servers/daily.0 ./servers/daily.0/file1 ...
4
votes
2answers
1k views

Rsync with ssh-keygen to ssh user with limited commands and specifc directory

I have two machines, local and remote. I want to back up my files and folders to the remote machine automatically, using rsync to transfer files and folders, using ssh-keygen to automatically log ...
4
votes
3answers
724 views

Backup with bash and rsync

Is there a way to auto-rename an existing file on the receiver? For example: if filename already exists, it auto-rename filename to something like filename_001, filename_002 and so on.... So far all ...
4
votes
2answers
6k views

Can I use rsync to create a list of only changed files?

I am using rsync in a bash script to keep files in sync between a few servers and a NAS. One issue I have run into is trying to generate a list of the files that have changed from the during the ...
4
votes
3answers
6k views

Escaping spaces in a remote path when using rsync over a remote SSH connection

When using SSH to connect rsync to a remote server, how do you escape spaces and such in the remote path? A simple backslash escapes the space for the local bash prompt, but on the remote machine the ...