Rsync is a fast and extraordinarily versatile file copying tool. It can copy locally, to/from another host over any remote shell, or to/from a remote rsync daemon

learn more… | top users | synonyms

6
votes
6answers
2k views

Remote file copy util (like rsync) but that will take account of data already copied (in this session)?

Let's say I have a directory with 2 files, both are identical and quite large (e.g. 2GB ea.) I want to rsync that directory to a remote host. As I understand it (and I could be wrong), rsync ...
6
votes
3answers
7k views

How to fix corrupt packet error for with rsync for (relatively) large files?

Trying to update files on a server, with the rsync command: rsync -ravq -e "ssh -o ConnectTimeout=2 -o ServerAliveInterval=2 -ServerAliveCountMax=2" --delete ./local_dir user@$SERVER:/dest_dir ...
6
votes
2answers
461 views

rsync for backup considered dangerous?

In our research group, we have the need to backup data acquired on the MRI scanner in a way that will preserve any scan ever acquired (even though data might be deleted from the scanner due to space ...
6
votes
2answers
3k views

Cron and rsync using file lock but is there a better way on Mac OS X Server?

I have this bash script and is run via cron on a regular interval: #!/bin/bash RSYNC=/usr/bin/rsync SSH=/usr/bin/ssh KEY=/Users/admin/Documents/Backup/rsync-key RUSER=philosophy RHOST=example.com ...
5
votes
7answers
12k views

Speeding up rsync over smb

I'm backing up a Linux box over SMB to a NAS. I mount the NAS locally and then I rsync a lot of data (100GB or so). I believe it's taking an awfully long time to do it: more than 12 hours. I would ...
5
votes
10answers
4k views

Using rsync to quickly upload a file that is similar to another file

I'm putting together a deployment script which tars up a directory of my code, names the tar file after the current date and time, pushes that up to the server, untars it in a directory of the same ...
5
votes
3answers
947 views

rsync: can't get rid of password prompt

I'm trying to run rsync to copy files (Centos 5.2 -> Centos 5.2) and I need run using nohup it so it can run after I disconnect my ssh session. The problem is I cannot get rsync to not prompt for a ...
5
votes
6answers
2k views

Is it possible to synchronize files between servers in real-time?

I have 5 or so webservers at a loadbalancer and if a user uploads a pic it can take up to a minute to rsync it accross (crontab). This can get frustrating for some thinking it failed. Is there a way ...
5
votes
6answers
11k views

Rsync over ssh: “ERROR: module is read only” suddenly appeared

I've used from some time rsync/ssh to backup my shared host contents to my personal Synology NAS (212j for that matter), and it worked quite well. For information, I use a password-less ssh ...
5
votes
3answers
6k views

Trying to setup chroot'd rsync

I'm trying to set up a backup server. I want to chroot each user (client) to its home directory, and only allow it to use sftp and rsync. I quickly discovered that I was not the only one trying to do ...
5
votes
1answer
5k views

Does rsync delete local files if they were deleted on source folder?

Here's what I'm trying to do: In essence, get files from one net storage device synced to another net storage device. The solution was to rsync from the source net storage to a local machine. Then ...
5
votes
6answers
7k views

rsync take too long to run

I have a load balancer setup involve 2 server. these 2 server mirror each other. the main usage of the blanacer is serving static files. Let's call them Server A and Server B. Server A will retrieve ...
5
votes
1answer
12k views

Optimizing rsync for transferring huge files quickly

This is what I need: Rsync daemon running on source server. Destination server does a rsync pull as below: rsync -ptv rsync://source_ip:document/source_path/*.abc destination_path/ Huge binary files ...
5
votes
1answer
871 views

Why does rsync have to be installed on both sides to work?

This might be a silly question, but I wanted to keep some of my servers clean and prefer not to install any additional software. But I always thought rsync would copy data over SSH and would not ...
5
votes
2answers
214 views

full or incremental backup of large number of files

I have a large amount of files in both quantity and total file size. (we're talking a few terabytes). I would like to sync these files/folders once to an external backup system, then run a daily task ...
5
votes
2answers
2k views

What is the default remote shell for rsync?

Just curious, what is the default remote shell used by rsync when you're not using it in daemon mode? According to the man page it can be run via rsh or ssh. However nowhere in the man page does it ...
5
votes
3answers
2k views

How to copy large (> 1 million) number of small files between two servers

I need to migrate about 1TB of data comprised of smaller files (most under 100KB) to another server. I've not even completely enumerated the files but estimates are between 1-2 million. The ...
5
votes
3answers
4k views

rsync and include / exclude. How hard can it be?

I'm trying to recursively copy a directory / file structure from one directory to another, keeping only html files. Should be a simple case of include / exclude shouldn't it? I just want to print out ...
5
votes
2answers
1k views

Equivalent of `nice` for filesystem IO?

When I'm running an rsync backup job which involves copying over large files, the machine running the backup (both Mac desktops and Linux servers) grinds to a halt and the load average goes through ...
5
votes
3answers
194 views

how to make rsync not uncompressing files?

I'm using rsync to copy files between servers, I'm using -z option what compresses data only for the network transfer stage. But I would like to leave compressed files on remote machine and I have ...
5
votes
1answer
1k views

does rsync's delta algorithm work on a samba share?

I have a cron job to use rsync to sync the data on a windows samba share (that i mount using /etc/fstab on boot). My problem is that the transfer seems to take almost as long as the initial transfer ...
5
votes
1answer
2k views

bind interface for upload : scp works, not rsync

I need to upload file on a specific interface. This cannot be set-up via iptables, because my script output on several interfaces. this is working fine : scp -oBindAddress=192.168.100.1 ... but ...
5
votes
1answer
757 views

Script launching 3 copies of rsync

I have a simple script that uses rsync to copy a Postgres database to a backup location for use with Point In Time Recovery. The script is run every 2 hours via a cron job for the postgres user. For ...
5
votes
2answers
1k views

Is rsync a good candidate for failover implementation (very large dataset)?

I have a large a set of data (+100 GB) which can be stored into files. Most of the files would be in the 5k-50k range (80%), then 50k - 500k (15%) and >500k (5%). The maximum expected size of a file ...
5
votes
5answers
1k views

Windows based rsync server

Is there a way to build a Windows server what can be target for other Windows file servers with rsync ?
5
votes
2answers
3k views

Malformed rsync command on ReadyNAS

I'm trying to set up rsync backups on my ReadyNAS and I'm getting the following error: ERROR: The remote path must start with a module name not a / This error is accompanied by the following ...
5
votes
7answers
2k views

fastest backup for large files

what is the fastest way to duplicate a folder contain few files with few hundred megs size and other few megs small files? (say, /var/lib/mysql) cp or tar or rsync or...??
5
votes
2answers
2k views

rsync ionice destination

To avoid rsync to starve the system and network we use ionice when starting rsync and set the --bwlimit parameter. For example: ionice -c2 -n7 rsync -aH --bwlimit=30000 /foo root@dest.com:/ This ...
5
votes
2answers
3k views

sshd_config ForceCommand /usr/bin/rsync error “connection unexpectedly closed”

How should ForceCommand be configured in order to allow the backup user to execute any rsync command? With ForceCommand set to /usr/bin/rsync I receive the error below: connection unexpectedly ...
5
votes
3answers
6k views

Using rsync from msysgit for binary files

I have three windows boxes that are connected to a linux box on my network in various ways (one's local to the network, one is a laptop and may or may not be local, and the other is my work desktop). ...
5
votes
2answers
892 views

Linux: Syncronize two directories permissions, users and groups and leave file contents untouched

In my Linux box I have two directories: work files with wrong permissions older versions of the same files with the right permissions (permissions and users and groups) I need to syncronize the ...
5
votes
1answer
2k views

rsync xattr oddness between mac & linux?

Hullo all - I'm using rsync to back up from my mac to a fileserver runnning linux, and I see the following error: rsync -e ssh -vaxE --delete --ignore-errors 192.168.1.3:/bkup/mac/Users/dave/ ...
5
votes
4answers
2k views

rsync/ssh/rsnapshot backup — remote script execution

I have the following settings in my configuration file: (/etc/rsnapshot.conf) Script/Tool: http://www.rsnapshot.org/ backup_script ssh root@host.domain.com "sh /home/user/backup_mysql.sh" ...
5
votes
1answer
510 views

Rsync + public key authentication security

I have read several articles how to automatically backup files with Rsync and public key authentication. All of them are very similar. I just finished setting up everything and everything works fine ...
5
votes
2answers
280 views

scp and rsync do not work (ssh works)

In my Debian Squeeze server, scp and rsync do not seem to copy over anything. Works fine with another server though (so not a client problem). When using scp, it just creates an empty file and waits. ...
5
votes
3answers
2k views

lsync, unison or some other inotify auto-syncing tool..?

I have an app farm which generates thumbnails and makes them available to a web server farm using nfs. However the performance is so poor so I am going to make a local copy of the files on each web ...
5
votes
1answer
704 views

Forward-sync to HDFS? (OR continue an incomplete hdfs upload?)

Anyone have a good suggestion for doing a forward sync to HDFS? ("forward-sync" in contrast to "bi-directional sync") Basically I have a large number of files I want to put into the HDFS. Its so ...
4
votes
7answers
12k views

Copying huge files between two remote machines - Efficiently

I have a shell script which keeps on copying huge files (2 GB to 5 GB) between remote systems. Key based authentication is used with agent-forwarding and everything works. For ex: Say the shell script ...
4
votes
5answers
5k views

How do I copy a directory tree but not the files in Linux?

I want to copy about 200 directories & subdirectories from one location to another but I don't want to copy the thousands of files within those directories. I am on Linux. Note: I don't have ...
4
votes
6answers
628 views

Is Rsync like subversion, but for a server?

I'm trying to learn how to use rsync. I want to create daily backs up of my production server. Right now I run the command rsync -azr /var/www/* www-data@backup.server.com:/var/www Now let's say ...
4
votes
5answers
5k views

Rsync Remote to Remote

I'm looking for a solution for using rsync between 2 remote servers. It seems like its not possible. Does any one know why its not possible? I ask that because I think if I know the reason, maybe I ...
4
votes
5answers
155 views

send files to dozens of servers quickly

We have a cluster of machines (around 50 and growing). Each machine has a search index that needs to be updated multiple times per day. We currently update the index on each machine individually, but ...
4
votes
2answers
2k views

rsync with --hard-links freezes

I have a large directory called servers, which contains many hard-links made by rsnapshot. That means that the structure is more or less like: ./servers ./servers/daily.0 ./servers/daily.0/file1 ...
4
votes
2answers
1k views

Rsync with ssh-keygen to ssh user with limited commands and specifc directory

I have two machines, local and remote. I want to back up my files and folders to the remote machine automatically, using rsync to transfer files and folders, using ssh-keygen to automatically log ...
4
votes
3answers
715 views

Backup with bash and rsync

Is there a way to auto-rename an existing file on the receiver? For example: if filename already exists, it auto-rename filename to something like filename_001, filename_002 and so on.... So far all ...
4
votes
2answers
5k views

Can I use rsync to create a list of only changed files?

I am using rsync in a bash script to keep files in sync between a few servers and a NAS. One issue I have run into is trying to generate a list of the files that have changed from the during the ...
4
votes
3answers
5k views

Escaping spaces in a remote path when using rsync over a remote SSH connection

When using SSH to connect rsync to a remote server, how do you escape spaces and such in the remote path? A simple backslash escapes the space for the local bash prompt, but on the remote machine the ...
4
votes
3answers
2k views

Using rsync to maintain a copy of a directory with a name that changes

I am using rsync on a linux system to synchronize a directory between the local disk and an attached USB drive. The problem I am experiencing is that the 3rd party system that creates the backup ...
4
votes
2answers
6k views

rsync - failed to set permission - operation not permitted

I got the following command with this error rsync -avz --links -O /home/jansiatest/.jenkins/workspace/svn_to_demo/trunk/CPS/ jansia@ps27670.dreamhost.com:/home/tasklite/temp sending incremental file ...
4
votes
2answers
7k views

Speed up rsync by running multiple instances at once?

I have to transfer a big directory to another server. My problem is that i cannot use the full bandwidth that would be available, because one concurrent tcp stream does not get as fast. Basically ...