Rsync is a fast and extraordinarily versatile file copying tool. It can copy locally, to/from another host over any remote shell, or to/from a remote rsync daemon

learn more… | top users | synonyms

5
votes
3answers
3k views

Backup strategy for eCryptfs

We have a small network with a Ubuntu server in the middle and Ubuntu notebooks around. The notebooks all use eCryptfs to encrypt the whole home directory of users. Usually, there are two or more ...
5
votes
3answers
12k views

Complex includes/excludes with rsync

I'm trying to work out the rsync filter syntax to perform complex include/excludes, and trying to achieve the following: Include / Exclude /home Include /home/user1/* Include /home/user2/subdir/* I'...
5
votes
7answers
2k views

fastest backup for large files

what is the fastest way to duplicate a folder contain few files with few hundred megs size and other few megs small files? (say, /var/lib/mysql) cp or tar or rsync or...??
5
votes
2answers
4k views

rsync --iconv option on Mac not working (sync from remote Linux server to local Mac)

I want to use rsync to backup data from a remote Linux server to my local Mac. And I want to initialise this operation on my local Mac. All works fine except that there is a special character problem: ...
5
votes
2answers
4k views

sshd_config ForceCommand /usr/bin/rsync error “connection unexpectedly closed”

How should ForceCommand be configured in order to allow the backup user to execute any rsync command? With ForceCommand set to /usr/bin/rsync I receive the error below: connection unexpectedly closed.....
5
votes
2answers
1k views

Linux: Syncronize two directories permissions, users and groups and leave file contents untouched

In my Linux box I have two directories: work files with wrong permissions older versions of the same files with the right permissions (permissions and users and groups) I need to syncronize the ...
5
votes
5answers
2k views

rsync/ssh/rsnapshot backup — remote script execution

I have the following settings in my configuration file: (/etc/rsnapshot.conf) Script/Tool: http://www.rsnapshot.org/ backup_script ssh root@host.domain.com "sh /home/user/backup_mysql.sh" ...
5
votes
1answer
822 views

Rsync + public key authentication security

I have read several articles how to automatically backup files with Rsync and public key authentication. All of them are very similar. I just finished setting up everything and everything works fine ...
5
votes
2answers
341 views

Cygwin rsyncd sends too many file names

I have rsyncd running as a Windows service, but when I try to access a module and sync from it, I get the files I want, as well as a lot that I do not — because they are not located in that module on ...
5
votes
2answers
387 views

scp and rsync do not work (ssh works)

In my Debian Squeeze server, scp and rsync do not seem to copy over anything. Works fine with another server though (so not a client problem). When using scp, it just creates an empty file and waits. /...
5
votes
3answers
3k views

lsync, unison or some other inotify auto-syncing tool..?

I have an app farm which generates thumbnails and makes them available to a web server farm using nfs. However the performance is so poor so I am going to make a local copy of the files on each web ...
5
votes
1answer
924 views

Forward-sync to HDFS? (OR continue an incomplete hdfs upload?)

Anyone have a good suggestion for doing a forward sync to HDFS? ("forward-sync" in contrast to "bi-directional sync") Basically I have a large number of files I want to put into the HDFS. Its so ...
5
votes
0answers
42 views

Can you scp, sftp, or rsync, a pipe? [migrated]

I want to tar and send a ~700GiB directory to a remote drive I don't control. I don't have the HDD space locally to create the tarball and then copy that over. The remote is also protected by rssh - ...
5
votes
0answers
99 views

BackupPC using rsync fails with “protocol version mismatch — is your shell clean?”

I just installed BackupPC in a Debian box to use it as backup server. As you know, this software makes use, among others, of rsync. When using the rsync option, it fails. Server: Debian Jessie. ...
4
votes
6answers
647 views

Is Rsync like subversion, but for a server?

I'm trying to learn how to use rsync. I want to create daily backs up of my production server. Right now I run the command rsync -azr /var/www/* www-data@backup.server.com:/var/www Now let's say ...
4
votes
5answers
175 views

send files to dozens of servers quickly

We have a cluster of machines (around 50 and growing). Each machine has a search index that needs to be updated multiple times per day. We currently update the index on each machine individually, but ...
4
votes
4answers
7k views

Can I rsync from Putty on windows to my Cent OS server?

I'm a bit new to the command line and UNIX but am finding it hugely useful. Is it possible for me to rsync from putty on my windows 7 machine to my virtual server? If not what are my other command ...
4
votes
2answers
1k views

Rsync with ssh-keygen to ssh user with limited commands and specifc directory

I have two machines, local and remote. I want to back up my files and folders to the remote machine automatically, using rsync to transfer files and folders, using ssh-keygen to automatically log ...
4
votes
2answers
5k views

Linux NTFS to NTFS rsync repeatedly recopying files

I wish to backup data mounted on an NTFS partition of a linux machine to the NTFS partition of another linux machine. The local NTFS drive is mounted with all files owned by root.root, but the remote ...
4
votes
3answers
791 views

Backup with bash and rsync

Is there a way to auto-rename an existing file on the receiver? For example: if filename already exists, it auto-rename filename to something like filename_001, filename_002 and so on.... So far all ...
4
votes
2answers
8k views

Speed up rsync by running multiple instances at once?

I have to transfer a big directory to another server. My problem is that i cannot use the full bandwidth that would be available, because one concurrent tcp stream does not get as fast. Basically ...
4
votes
3answers
2k views

Using rsync to maintain a copy of a directory with a name that changes

I am using rsync on a linux system to synchronize a directory between the local disk and an attached USB drive. The problem I am experiencing is that the 3rd party system that creates the backup ...
4
votes
1answer
3k views

Rsync failing with “file too large”

I am writing what should be a simple backup script to copy all of my media files to an external USB drive. There seems to be one file that it chokes on and I'm not sure why. Just trying to copy that ...
4
votes
1answer
3k views

Can't get rsync over sftp to work

I'm trying to set up a backup system from an Ubuntu server to a Synology NAS (DS413j) using rsync and sftp. I have created a user for this that we can call ubuntu-backup. I have a directory in ubuntu-...
4
votes
2answers
9k views

rsync - failed to set permission - operation not permitted

I got the following command with this error rsync -avz --links -O /home/jansiatest/.jenkins/workspace/svn_to_demo/trunk/CPS/ jansia@ps27670.dreamhost.com:/home/tasklite/temp sending incremental file ...
4
votes
3answers
5k views

How to delete empty source directories when moving folders with rsync?

rsync can be coaxed into moving folders very similarly and superiorly to traditional mv with the following options: rsync -axvvES --remove-source-files source_directory /destination/ However one ...
4
votes
2answers
9k views

ssh is ok, but connection refused when trying to rsync, why?

Editing the question, sorry for the confusion I need make cron job that will backup a directory that resides on a remote server, have the backup on the machine running the cron job. The remote server ...
4
votes
3answers
3k views

Chunking large rsync transfers?

We use rsync to update a mirror of our primary file server to an off-site colocated backup server. One of the issues we currently have is that our file server has > 1TB of mostly smaller files (in ...
4
votes
2answers
4k views

Using rsync to rename files during copying with --files-from?

Using rsync, how can I rename files when copying with the --files-from argument? I have about 190,000 files, each of which need to be renamed when copying from source to destination. I plan to have ...
4
votes
1answer
1k views

rsync copies unmodified files between different file systems

I try to synchronize files between two machines using rsync, but it seems to copy all the files regardless file modification time. Two consecutive call of the command produces the same results, so it ...
4
votes
2answers
117 views

executing a command again once it stops (rsync, FreeBSD)

I started doing backups via rsync to a remote server. This works all fine and good but 40 gig take their time and as I get a dynamic IP it gets interrupted every day. So every morning when I wake up I ...
4
votes
3answers
2k views

Forcing rsync to convert file names to lower case

We are using rsync to transfer some (millions) files from a Windows (NTFS/CYGWIN) server to a Linux (RHEL) server. We would like to force all file and directory names on the Linux box to be lower ...
4
votes
2answers
6k views

Freenas — Choosing the right redundant storage regime (RAID, ZFS, Rsync?)

I've been running a freenas box for a few months as a proof of concept and all seems to be well now reliability wise. So it's time to up the storage capacity and build it into production mode. I was ...
4
votes
3answers
531 views

rsync many small files with long filenames takes a lot of bandwidth

I have a file storage server which stores files on disk using the file's sha256 hash as the filename, along with the file extension, and in three levels of directories, e.g. a PDF file with sha256 ...
4
votes
1answer
634 views

Preserving existing destination symlinks with rsync

I am using rsync to keep a local debian repository updated. Recently, the disk I am using to store it started running low on space, so I decided to use symlinks to allow me to move some of the ...
4
votes
3answers
3k views

Rsync failing when trying to run via cron

I am trying to set up a cron to run this command: rsync -e "ssh -i /home/ldsmp3/ssh-key -o GSSAPIAuthentication=no" -az /home/ldsmp3/public_html s3299@farm.s3rsync.com:remote-dir%%bak-ldsmp3.com%%...
4
votes
3answers
3k views

rsync and --remove-source-files

I use rsync v. 3.0.4 and when I need to move something I use it with the --remove-source-files. I prefer rsync than mv. Unfortunately, when I use --remove-source-files, the directories are left on ...
4
votes
3answers
4k views

VMware ESXi running on dual NFS NAS Configuration using rsync and snapshot scheduling [closed]

I'm considering a setup for a client. GOALS 10 users 2 servers maximum redundance (with minimal costs) maximum scalability (with minimal costs) HARDWARE Here's the hardware I'm considering: 2 x Dell ...
4
votes
2answers
3k views

rsync multiple source directories to multiple dest directories

I have multiple source dirs, say /home/me/source1 and /mnt/somedisk/source2. I want to push them both to something like /home/someoneelse/dest1 and /home/someoneelse/dest respectively. I'd like to ...
4
votes
1answer
808 views

how to backup svn remotely without a local copy

I have a subversion server that is running short on disk. I'd like to archive one of the larger repositories to free up some space, so I need to make a full backup of the repository files on a remote ...
4
votes
2answers
3k views

rsync via multiple ssh connections with a key

I need to do a rsync from a target server with a server in the middle and I have the same public key on both the target and middle server. I want to use that key so I don't have to do password ...
4
votes
2answers
758 views

synchronizing storage servers

we have a storage server, with currently an amount of about 20TB of media files which we want to synchronize with a second storage server, for backup and fail-over. the facts are: we're storing ...
4
votes
2answers
2k views

Multi Thread Rsync Transfer

For some reason when running a single rsync command I am getting 1 MB/sec to 2 MB/sec even when I connecting 2 servers both connected to 1 Gbps ports. rsync -v --progress -e ssh /backup/mysqldata/...
4
votes
2answers
6k views

RSync over SSH - permission denied even though the user is in the root group

I have a need to copy files between servers through the web. I'm using RSYNC over ssh to do so. The problem is, I need to be able to transfer files, no matter where the files is. I created a user ...
4
votes
1answer
2k views

Linux server sync to an Amazon S3 bucket

I am looking for a stable solution to replace a classic server backup to another server using rsync. I have to sync a whole filesystem (more than 1Tb) to Amazon S3. Where am I? Solution 1: I mapped ...
4
votes
1answer
223 views

Trying to script rsync using pam_exec

I'm trying to write a bash script that will execute rsync when called by pam_exec. I've tried a couple different ways, and I'm not sure what I'm doing wrong. When I try to run the script at login by ...
4
votes
2answers
954 views

rsync not removing single quotes around array of strings --filter list

Trying to safely retrieve files from heavily infected Windows XP and 2003 machines that came to us from an acquisition. I have a similar need for automating a script that would run on PXE booted ...
4
votes
3answers
287 views

Backup Permissions for Entire File System

Here's the skinny: I have a Linode VPS that I want to backup to my local laptop via rsync. I have just generated ssh keys for the communication and left a blank passphrase for the private key so I can ...
4
votes
1answer
404 views

Rsnapshot hard-link ONLY + diffs

We have a 39 TB linux server (mdadm raid 6 15 3TB Drives) that has about 12TB of working data on it. Initially I wanted to create a snapshot system that only saved the data that was changed, much like ...
4
votes
5answers
14k views

rsync --files-from or --include-from

I tried the command: rsync -v --include-from=/path/to/list.lst /home/user /path/to/backup list.lst contains for example .gnupg/ .pki/ .gnome2/keyrings/ .mozilla/firefox/*.default/...