Questions tagged [rsync]
Rsync is a fast and extraordinarily versatile file copying tool. It can copy locally, to/from another host over any remote shell, or to/from a remote rsync daemon
1,613
questions
14
votes
3
answers
41k
views
Set maximum speed for scp
Is there a way to limit the speed of scp (or rsync)? I couldn't find any information in the man page, nor on the internet.
- 586
14
votes
1
answer
23k
views
Force rsync to copy symlink, even if referent does not exist
I am getting this message with rsync:
symlink has no referent: "/a/path"
Well, rsync is right: in the source filesystem, the (symbolic) link is pointing to a non-existent location. That is intended. ...
- 4,350
14
votes
1
answer
6k
views
Why does rsync have to be installed on both sides to work?
This might be a silly question, but I wanted to keep some of my servers clean and prefer not to install any additional software.
But I always thought rsync would copy data over SSH and would not ...
- 4,033
14
votes
1
answer
18k
views
Rsync failing with "file too large"
I am writing what should be a simple backup script to copy all of my media files to an external USB drive. There seems to be one file that it chokes on and I'm not sure why. Just trying to copy that ...
- 187
14
votes
1
answer
7k
views
Preserving existing destination symlinks with rsync
I am using rsync(1) to keep a local debian repository updated. Recently, the disk I am using to store it started running low on space, so I decided to use symlinks to allow me to move some of the ...
- 243
13
votes
4
answers
19k
views
Why don't people just use rsync to back up vmware guests?
If I am running a modern vmware ESXi system, I can drop in a statically linked rsync binary and rsync files to any destination over SSH.
I'm trying to understand why most (all ?) backup of vmware ...
- 221
13
votes
8
answers
18k
views
make a cronjob wait for previous rsync job to finish
I'm using rsync to backup some data from one server to another. All works fine, but it might take longer to finish depending on how much data there is to transfer.
Is there any gauranteed way to ...
- 340
13
votes
3
answers
15k
views
rsync --iconv option on Mac not working (sync from remote Linux server to local Mac)
I want to use rsync to backup data from a remote Linux server to my local Mac. And I want to initialise this operation on my local Mac. All works fine except that there is a special character problem: ...
- 441
13
votes
3
answers
39k
views
How to add timestamp and file list to rsync log?
I want to add both timestamp and file list to the log. Currently, I can only get one. The following command to can track the updated file list but it does not add timestamp.
rsync -avz --progress --...
- 2,418
13
votes
4
answers
16k
views
Escaping spaces in a remote path when using rsync over a remote SSH connection
When using SSH to connect rsync to a remote server, how do you escape spaces and such in the remote path? A simple backslash escapes the space for the local bash prompt, but on the remote machine the ...
- 275
13
votes
5
answers
46k
views
How to fix corrupt packet error for with rsync for (relatively) large files?
Trying to update files on a server, with the rsync command:
rsync -ravq -e "ssh -o ConnectTimeout=2 -o ServerAliveInterval=2 -ServerAliveCountMax=2" --delete ./local_dir user@$SERVER:/dest_dir
...
- 677
13
votes
4
answers
11k
views
How to cleanup after failed rsync?
I killed a local "rsync -ah" process.
It left junk files like ".abc.vmdk.zxivMN" in the target directory.
The junk files do not get deleted on consecutive "rsync -ah" runs.
Details:
I ran diff on ...
- 259
13
votes
4
answers
6k
views
Tell Rsync to Skip Current file During Transfer?
Is there a way to tell rsync to skip its current file while a sync is in progress, maybe by sending it a particular signal?
I already know about ignoring based on patterns, but this would be handy to ...
- 84.4k
13
votes
9
answers
16k
views
Windows rsync that supports long file names, or an good alternative?
I currently rsync on a Linux host to copy things from Windows to my Linux box. But I can't copy files with long names.
I have tried DeltaCopy, cwrsync, and cygwin. From what I have found there all ...
- 132k
13
votes
1
answer
808
views
Does rsync change file content (dealing with hardlinks)?
Imagine I have a file remote/A which syncs to local/A and local/B is a hardlink of local/A. Is there a threat that on a next remote/A change and synchronization rsync will not delete local/A but only ...
- 232
13
votes
4
answers
17k
views
rsync to compressed archive of file
I take backups of a server with rsync, but those backups are quite large, and I'd like to compress them. Is there a way to have some sort of wrapper around rsync to gunzip the file in the backup, ...
- 2,331
13
votes
1
answer
27k
views
rsync requires rsync-path when remote rsync is in path
I am quite confused as to why rsync requires --rsync-path flag even when remote rsync is in path.
Consider:
$ rsync -avze 'ssh -p 22' --delete public/ [email protected]:~/public_html
bash: /usr/...
- 299
12
votes
2
answers
41k
views
rsync - failed to set permission - operation not permitted
I got the following command with this error
rsync -avz --links -O /home/jansiatest/.jenkins/workspace/svn_to_demo/trunk/CPS/ [email protected]:/home/tasklite/temp
sending incremental file ...
- 261
12
votes
1
answer
6k
views
Getting rsync to delete files from the *sender* after transfer
I'm using rsync to move one directory to another. Is there some way to get rsync to incrementally delete a file from the sender after it has copied it to the reciever?
- 32.3k
12
votes
2
answers
2k
views
Inode tables shrinking sharply over time causing rsync/inode problems
We setup a Linux (it's on Amazon AWS, a CentOS-like system although we are not exactly sure the customizations done on it) system with 4TB storage as an XFS volume over LVM (ultimately to be used for ...
- 121
11
votes
7
answers
33k
views
Speeding up rsync over smb
I'm backing up a Linux box over SMB to a NAS. I mount the NAS locally and then I rsync a lot of data (100GB or so). I believe it's taking an awfully long time to do it: more than 12 hours. I would ...
- 7,508
11
votes
2
answers
29k
views
Run totally silent rsync?
I have a cronjob that runs hourly, and is totally silent unless something goes wrong. Well ... almost ...
A part of the job is
rsync --del -Cacqrz public/. [email protected]:/target/path
This ...
- 219
11
votes
5
answers
18k
views
ZFS Sync over unreliable, slow WAN. ZFS replication, or rsync?
I've been tasked with making an off-site backup work over the WAN. Both storage boxes are FreeBSD based NAS boxes running ZFS.
Once or twice a week, 15-60 gigs of photography data gets dumped to the ...
- 1,229
11
votes
4
answers
4k
views
Deploying Files To Multiple Servers
We have a central repository of files in say server-1 at /srv/www. Then we have N number of servers as well. We want server-1 to be able to deploy its files in /srv/www to all servers as quickly and ...
- 5,438
11
votes
3
answers
28k
views
Preserving file and folder permissions with rsync
I maintain a backup of my email accounts using this command:
sudo rsync -av --delete --progress -e "ssh -p pNumber" --rsync-path="/usr/bin/rsync" /vmail/ user@my_backup_server:/home/user/backups/...
- 214
11
votes
1
answer
39k
views
Does rsync overwrite the new file
i use rsync -avh
I just want to know that if the file on destination already exists does the rsync overwrite the new file or not
John
11
votes
2
answers
20k
views
Can't get rsync to work in daemon-over-ssh mode
I'm trying to set up rsync to copy the data from a server every day. In order to make the system as restricted as possible, I'm trying to use the mode described in the man page as:
"USING RSYNC-DAEMON ...
- 2,331
11
votes
4
answers
4k
views
rsync directory so all changes appear atomically
I do some nightly and weekly mirrors of frequently used repositories for the local network. On a few occasions, someone has tried to do an update while the rsync is happening and failed because the ...
- 420
11
votes
3
answers
8k
views
Has anyone achieved true differential sync with rsync in ESXi?
Berate me later on the fact that I'm using the service console to do anything in ESXi...
I've got a working rsync binary (v3.0.4) that I can use in ESXi 4.1U1. I tend to use rsync over cp when ...
- 135
11
votes
2
answers
2k
views
Equivalent of `nice` for filesystem IO?
When I'm running an rsync backup job which involves copying over large files, the machine running the backup (both Mac desktops and Linux servers) grinds to a halt and the load average goes through ...
- 2,277
11
votes
1
answer
4k
views
Why does rsync forks itself? And why one such forked process is almost kinda idle (as seen in iotop)?
This is referring to the matter stated here and I am also experiencing the same.
In one of my servers, I ran an rsync, to backup a huge directory (size greater than 300 Gb) to a different disk, ...
- 295
11
votes
3
answers
3k
views
How to mirror filesystems with millions of hardlinks?
We have one big problem at the moment: We need to mirror a filesystem for one of our customers. Thats usual not really a problem, but here it is:
On this filesystem there is one folder with millions ...
- 1,710
10
votes
7
answers
16k
views
what is the fastest and most reliable way of transferring a lot of files?
I'm trying to transfer about 100k files totaling 90gb. Right now I'm using rsync daemon but its slow 3.4mb/s and I need to do this a number of times. I'm wondering what options do I have that would ...
- 945
10
votes
2
answers
400
views
Server migration: most efficient way
I have been tasked with migrating one of our sites between servers (two different hosts). Both environments are linux.
The site streams video, so the server is currently filled with media files (...
- 103
10
votes
3
answers
15k
views
is it safe to running multiple instances of rsync together
I'm wondering if its safe to use multiple rsync instances with the same command line to backup the same directory on the local server to the same directory on the remote server? basically if I ran the ...
- 945
10
votes
2
answers
6k
views
rsync from linux host to fat32
i like to backup my linux based NAS (small files like picutres) to an attached USB Disk with FAT32.
For this i like to use rsync.
I know, that this will not work correct per default. rsync will sync ...
- 294
10
votes
1
answer
6k
views
Using a shell script to check if rsync made any changes
I've got a bash script that rsync's two directories. Sometimes there's a change, odds are there's nothing.
I want to run a command only if rsync actually made a change (add/update a file). ...
- 141
10
votes
4
answers
7k
views
How to make sshfs + VPN + git a tolerable working environment?
Currently the code base for the project I am working on is remotely on a company server. and it has to stay like that. also the remote git repository cannot be made public.
My current setup is:
...
- 209
10
votes
1
answer
4k
views
rsync xattr oddness between mac & linux?
Hullo all - I'm using rsync to back up from my mac to a fileserver runnning linux, and I see the following error:
rsync -e ssh -vaxE --delete --ignore-errors 192.168.1.3:/bkup/mac/Users/dave/ /Users/...
- 203
10
votes
2
answers
16k
views
rsync copy new files only, no updates?
I would like to use rsync to copy files from a remote server. I would like to
only copy files from server, not push any back out.
copy files that don't exist. If the remote file is newer I do not ...
user274
9
votes
5
answers
18k
views
Should I use rsync compression over a gigabit LAN?
I normally use rsync's -z option to enable compression when transferring files over the internet. However, if I am on my own (idle) gigabit LAN, do I still want compression? Or will it be faster ...
- 193
9
votes
13
answers
62k
views
rsync error: error in rsync protocol data stream (code 12) at io.c(600) [sender=3.0.6]
Recently I have been unable to rsync over ssh. Each time I get the same error
bash: rsync: command not found
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: ...
- 113
9
votes
4
answers
5k
views
How can I easily confirm in Linux that two separate directories have the exact same contents?
CentOS 5.x
Mq question seemed similar to this one but I wasn't sure...
I have two servers (completely isolated from each other), each with a directory and sub-directories that should have the same ...
- 12k
9
votes
2
answers
33k
views
rsync "Operation not permitted"
I am trying to so a simple rsync to a USB hard disc, the command is
rsync -az --modify-window=2 /home /mnt/exthd/tmp/
I am running this as root.
but I get a lot of errors that all seem to relate ...
9
votes
3
answers
36k
views
rsync via cron. How do I enable logging?
I'm backing up a remote server to another computer using rsync.
In cron.daily I have a file with this:
rsync -avz -e ssh [email protected]:/ /mybackup/
It uses a public / private key pair to login. ...
- 315
9
votes
4
answers
3k
views
Linux to Linux, 10 TB transfer?
I've looked at all the previous similar questions, but the answers seemed to be all over the place and no one was moving a lot of data (100 GB is different from 10 TB).
I've got about 10&...
- 193
9
votes
1
answer
8k
views
Does rsync delete local files if they were deleted on source folder?
Here's what I'm trying to do:
In essence, get files from one net storage device synced to another net storage device.
The solution was to rsync from the source net storage to a local machine. Then ...
- 236
9
votes
3
answers
10k
views
rsync: How to exclude Dotfiles only in topmost directory?
When performing backups under Bash with rsync, I'm trying to exclude all dotfiles and hidden directories in the top directory, but not those in otherwise targeted directories. For example:
/copyme.c ...
user937189
9
votes
4
answers
35k
views
rsync error: remote command not found (code 127) at io.c(605):
I get the following error:
sudo ionice -c 3 nice -n +19 rsync -av --progress -e 'ionice -c 3 nice -n +19 ssh -l root -p 22 192.168.0.1' 192.168.0.1:/domains/remote/. /domains/local/;
[email protected]....
- 354
9
votes
1
answer
12k
views
bind interface for upload : scp works, not rsync
I need to upload file on a specific interface.
This cannot be set-up via iptables, because my script output on several interfaces.
this is working fine :
scp -oBindAddress=192.168.100.1 ...
but
...
- 497