Rsync is a fast and extraordinarily versatile file copying tool. It can copy locally, to/from another host over any remote shell, or to/from a remote rsync daemon

learn more… | top users | synonyms

11
votes
4answers
6k views

Converting UTF-8 NFD filenames to UTF-8 NFC, in either rsync or afpd

I have a home file server running FreeNAS 8. A few days ago I used rsync to upload my entire iTunes library from Mac so that I could load my library over the network instead of off a slow USB drive. ...
11
votes
8answers
17k views

How do you synchronise huge sparse files (VM disk images) between machines?

Is there a command, such as rsync, which can synchronise huge, sparse, files from one linux server to another? It is very important that the destination file remains sparse. It may be longer (but ...
11
votes
2answers
3k views

Can I make rsync output summary only?

Because I use rsync to backup a directory which is very big, containing many sub-dirs and files, so I dont want to see the "incremental file list", I want to know the summary in the end only. but if I ...
11
votes
5answers
5k views

Synchronizing very large folder structures

We have a folder structure on our intranet which contains around 800,000 files divvied up into around 4,000 folders. We need to synchronize this to a small cluster of machines in our DMZs. The depth ...
11
votes
3answers
13k views

Maximizing rsync performance and throughput - directly-connected gigabit servers

I have two Dell R515 servers running CentOS 6.5, with one of the broadcom NICs in each directly attached to the other. I use the direct link to push backups from the main server in the pair to the ...
11
votes
2answers
3k views

Which is better for website backup - rsync or git push

I run 2 LAMP web servers at different providers for disaster recovery purposes - a high powered live server, and a low powered backup server. Currently I rsync all the data from the live server to ...
10
votes
6answers
8k views

what is the fastest and most reliable way of transferring a lot of files?

I'm trying to transfer about 100k files totaling 90gb. Right now I'm using rsync daemon but its slow 3.4mb/s and I need to do this a number of times. I'm wondering what options do I have that would ...
10
votes
4answers
2k views

Deploying Files To Multiple Servers

We have a central repository of files in say server-1 at /srv/www. Then we have N number of servers as well. We want server-1 to be able to deploy its files in /srv/www to all servers as quickly and ...
10
votes
3answers
19k views

run rsync as root but keep user ownership

I would like to backup user files from one server to another with rsync. but I noticed that the user folders change to root. how can I keep the user permissions with rsync (running by root)?
10
votes
4answers
27k views

For large files compress first then transfer or rsync -z? which would be fastest?

I have a ton of relativity small data files but they take up about 50 GB and I need them transferred to a different machine. I was trying to think of the most efficient way to do this. Thoughts I had ...
10
votes
2answers
2k views

How do I use rsync with an IPv6 address via SSH?

I've been having difficulty with connecting to my IPv6 address via rsync. Because the argument for the destination folder is colon-separated, the IPv6 address disrupts this like so: ...
10
votes
2answers
329 views

Server migration: most efficient way

I have been tasked with migrating one of our sites between servers (two different hosts). Both environments are linux. The site streams video, so the server is currently filled with media files ...
10
votes
2answers
4k views

Can rsync display current average speed?

When I'm transferring large quantities of data using rsync, it would be helpful if I could have the average speed up until now at a glance, rather than a bunch of different speeds for each file.
10
votes
4answers
13k views

How to speed up rsync for small files

I'm trying to transfer thousands of small files from one server to another using the following command: rsync -zr --delete /home/user/ user@10.1.1.1::backup Currently the transfer takes a long time ...
10
votes
1answer
3k views

Why does rsync spawn multiple processes for me?

I am using the following cron statement to backup from one folder to another folder in the same machine: 19 21 * * * root rsync -ac --delete /source/folder /dest/folder When I use pstree, I see ...
10
votes
3answers
7k views

Has anyone achieved true differential sync with rsync in ESXi?

Berate me later on the fact that I'm using the service console to do anything in ESXi... I've got a working rsync binary (v3.0.4) that I can use in ESXi 4.1U1. I tend to use rsync over cp when ...
10
votes
2answers
12k views

rsync copy new files only, no updates?

I would like to use rsync to copy files from a remote server. I would like to only copy files from server, not push any back out. copy files that don't exist. If the remote file is newer I do not ...
10
votes
3answers
1k views

How to mirror filesystems with millions of hardlinks?

We have one big problem at the moment: We need to mirror a filesystem for one of our customers. Thats usual not really a problem, but here it is: On this filesystem there is one folder with millions ...
9
votes
4answers
4k views

Do you need -e ssh for rsync?

I always used to use the following command when copying from a server: rsync --progress -avze ssh user@host:/path/to/files ./here. However, a friend of mind showed me that I can simply do rsync ...
9
votes
4answers
2k views

Linux to Linux, 10 TB transfer?

I've looked at all the previous similar questions, but the answers seemed to be all over the place and no one was moving a lot of data (100 GB is different from 10 TB). I've got about ...
9
votes
3answers
9k views

Rsync seems incompatible with .bashrc (causes “is your shell clean?”)

It turns out rsync can't work with a remote server which has a .bashrc file? At local client i got when run rsync: protocol version mismatch -- is your shell clean? (see the rsync man page for an ...
9
votes
2answers
10k views

How to add timestamp and file list to rsync log?

I want to add both timestamp and file list to the log. Currently, I can only get one. The following command to can track the updated file list but it does not add timestamp. rsync -avz --progress ...
9
votes
1answer
10k views

What exactly will --delete-excluded do for rsync?

I use rsync with great pleasure for backing up my servers, although I do have a question about the --delete-excluded parameter. /usr/bin/rsync -a --delete --numeric-ids --relative --delete-excluded ...
8
votes
3answers
6k views

Why don't people just use rsync to back up vmware guests?

If I am running a modern vmware ESXi system, I can drop in a statically linked rsync binary and rsync files to any destination over SSH. I'm trying to understand why most (all ?) backup of vmware ...
8
votes
10answers
27k views

Using Rsync to backup to an external drive

I'm buying an external hard drive to back up the computers in my house (finally!!). I'm hoping to use rsync. I've seen an example that does (or seems to do) exactly what I want. Something like ...
8
votes
5answers
11k views

How does RSYNC does incremental Backups

How does RSYNC knows which files are chnages and which are not. Does it logs its data anywhere in the file. Because I want to do incrrmental backups but first it will transfer all files. So my main ...
8
votes
5answers
12k views

ZFS Sync over unreliable, slow WAN. ZFS replication, or rsync?

I've been tasked with making an off-site backup work over the WAN. Both storage boxes are FreeBSD based NAS boxes running ZFS. Once or twice a week, 15-60 gigs of photography data gets dumped to the ...
8
votes
4answers
7k views

Encrypted Remote Backups via Rsync?

I run a small modest CentOS server at my house for misc backups, etc. My friend on the other side of town also runs a small modest server at his house for similar purposes. We have been toying with ...
8
votes
2answers
9k views

Can't get rsync to work in daemon-over-ssh mode

I'm trying to set up rsync to copy the data from a server every day. In order to make the system as restricted as possible, I'm trying to use the mode described in the man page as: "USING RSYNC-DAEMON ...
8
votes
1answer
506 views

How to cleanup after failed rsync?

I killed a local "rsync -ah" process. It left junk files like ".abc.vmdk.zxivMN" in the target directory. The junk files do not get deleted on consecutive "rsync -ah" runs. Details: I ran diff on ...
8
votes
6answers
29k views

Is there any other reason for “no space left on device”?

I am using Dirvish on a Ubuntu server system for backing up a hd to an external usb 3.0 drive. Until a few days ago, everything worked fine, but now every backup fails with "no space left on device ...
8
votes
4answers
14k views

Multi Threaded file sync between 2 Linux servers

At the moment i'm running rsync for 2.2 million files total of 250GB and that just takes ages 700K files in 6 hours. Does anyone know a rsync like tool that can do this with multiple threads so it ...
7
votes
8answers
4k views

make a cronjob wait for previous rsync job to finish

I'm using rsync to backup some data from one server to another. All works fine, but it might take longer to finish depending on how much data there is to transfer. Is there any gauranteed way to ...
7
votes
4answers
2k views

How can I easily confirm in Linux that two separate directories have the exact same contents?

CentOS 5.x Mq question seemed similar to this one but I wasn't sure... I have two servers (completely isolated from each other), each with a directory and sub-directories that should have the same ...
7
votes
6answers
7k views

How do I copy a directory tree but not the files in Linux?

I want to copy about 200 directories & subdirectories from one location to another but I don't want to copy the thousands of files within those directories. I am on Linux. Note: I don't have ...
7
votes
9answers
1k views

Copy 10 million images in a single folder to another server

Now I know you shouldn't ever put 10 million files into a single directory to begin with. Blame it on the developers, but as it stands that’s where I am at. We will be fixing it and moving them into ...
7
votes
6answers
14k views

Rsync over ssh: “ERROR: module is read only” suddenly appeared

I've used from some time rsync/ssh to backup my shared host contents to my personal Synology NAS (212j for that matter), and it worked quite well. For information, I use a password-less ssh ...
7
votes
6answers
6k views

improving rsync backup performance

What are the best techniques to improve rsync over ssh mirroring between unix boxes, assuming that one system will always have the master copy and the other system will always have a recent copy (less ...
7
votes
2answers
9k views

rsync : transferring files in one direction only

Is it possible to use rsync to copy files in one direction only? For example, suppose we have: left/a.txt right/a.txt where the files are initially identical. If one then modifies right/a.txt, ...
7
votes
3answers
16k views

rsync via cron. How do I enable logging?

I'm backing up a remote server to another computer using rsync. In cron.daily I have a file with this: rsync -avz -e ssh root@example.com:/ /mybackup/ It uses a public / private key pair to login. ...
7
votes
5answers
4k views

Forcing rsync to non-interactive mode

I would like to use rsync within a python script. I'm calling it using the subprocess module, and authenticating using public keys stored at the authorized_key file on the remote machine. The only ...
7
votes
1answer
5k views

Does rsync delete local files if they were deleted on source folder?

Here's what I'm trying to do: In essence, get files from one net storage device synced to another net storage device. The solution was to rsync from the source net storage to a local machine. Then ...
7
votes
4answers
6k views

How can I reduce resource usage when copying a large file?

I need to move a large file (corrupt MySQL table ~40GB) onto a seperate server in order to repair it. (When trying to repair on my production server, it quickly killed the server). In order to do ...
7
votes
3answers
9k views

Which is best for backups rsync vs rdiff vs rsnapshot

I am using rsync for backups from remote FTP to local computer. I read on internet that rsnapshot is better. Just want to know which is used in production environment
7
votes
2answers
2k views

Using rsync to only delete extraneous files

What's the best way of comparing two directory structures and deleting extraneous files and directories in the target location? I have a small web photo gallery app that I'm developing. Users add ...
7
votes
2answers
3k views

Copy directory tree without empty directories?

I have the following tree # upper letters = directory # lower letters = files A |-- B |-- C |-- D |-- e <= file |-- F |-- G I need to copy this tree to another destination, ...
7
votes
3answers
2k views

Tell Rsync to Skip Current file During Transfer?

Is there a way to tell rsync to skip its current file while a sync is in progress, maybe by sending it a particular signal? I already know about ignoring based on patterns, but this would be handy to ...
7
votes
4answers
6k views

rsync to compressed archive of file

I take backups of a server with rsync, but those backups are quite large, and I'd like to compress them. Is there a way to have some sort of wrapper around rsync to gunzip the file in the backup, ...
7
votes
4answers
6k views

Rsync -avzHP follows hardlinks instead of copying them as hardlinks

I use rsnapshot to create hourly/daily/weekly/monthly backups of my "work"-share. Now I'm trying to copy the whole backup-directory onto an external drive using rsync. I used this command/parameters ...
7
votes
1answer
6k views

How can I prevent rsync replacing symbolic links with new directories in the target directory hierarchy?

I am using rsync to synchronize files between a source area and a target area, where the target directory hierarchy contains symbolic links. When a source directory has the same name and relative ...