Questions tagged [rsync]
Rsync is a fast and extraordinarily versatile file copying tool. It can copy locally, to/from another host over any remote shell, or to/from a remote rsync daemon
307
questions with no upvoted or accepted answers
7
votes
0
answers
1k
views
How to re-compile iconv on Linux (Ubuntu 14.04 LTS) with pseudo-charset UTF8-MAC?
I was googling around for hours to find a solution for my problem and I couldn't get it to work:
I have to rsync a file structure on an ext4 formatted drive to a hfs+ formatted drive. Folders and ...
- 166
7
votes
0
answers
2k
views
How can I let rsync display both total and partial progress?
I've found that rsync's total progress is (somewhat) shown with --info=progress2, while partial progress (i.e. the current file) is shown with --info=progress (or -P).
Is there a way to show both?
I'...
- 171
4
votes
0
answers
2k
views
why `rsync --sparse` create much larger file than the source?
I try to rsync between 2 local disks, thus there is no compression. the sparse file has apparent size of 50GB, actual size of 16GB. I tried rsync -a -P --sparse, it did generate sparse file on ...
- 322
4
votes
1
answer
550
views
Rsync takes 8+ hours to backup 15M+ files
I have ~700GB storage where I store ~15M files, hence average file size is ~50KB. To back it up, over night I run a simple rsync script with the following set of flags:
--archive --update --compress -...
- 378
4
votes
0
answers
5k
views
DeltaCopy (Rsync for windows) giving an error on task run
I was suggested DeltaCopy to set up a nightly backup between two Windows 2008 R2 servers.
After setting up the server side (Amazon EC2 instance where files will be backed to) and setting up the ...
- 277
4
votes
2
answers
618
views
RSync backup of Subversion repository with Rsyncrypto
This is a crosspost from stackoverflow because I find that I might get a much qualified audience here
The Situation
I have a pretty large Subversion repository that I am trying to backup efficiently....
- 161
3
votes
0
answers
798
views
Getting lsetxattr errors using rsync with some files that have Apple resource forks
I'm running a backup of a MacOS machine to a Linux server using rsync. There are a few files that give errors every time, and I'm trying to figure out how to deal with them. I'd like to not get these ...
3
votes
0
answers
2k
views
vagrant ssh - asks for password after adding synced_folder
I can vagrant ssh into my box fine
Welcome to Ubuntu 14.04.2 LTS (GNU/Linux 3.16.0-30-generic x86_64)
* Documentation: https://help.ubuntu.com/
Last login: Mon Apr 13 13:05:26 2015 from 10.0.2.2
...
- 127
3
votes
0
answers
525
views
Solution for file data backup, versioning and replication
We have a growing set of data files (.wav files, image files etc) which are data, i.e not part of the application code - uploaded and modified by users. The number of files is in the 1000s and the ...
- 237
3
votes
1
answer
2k
views
Synology NAS - rsync messing up versioning / deduplication
Is it true that Synology DSM 4.3's default rsync implementation is not able to handle "vast" amounts of data and could mess up versioning / deduplication? Could it be that any of the variables (see ...
- 141
3
votes
1
answer
761
views
rsync stop before filesystem full
I would like to implement a rsync that should stop if the destination filesystem free space goes down to less than 10G and I wonder what is the best way to do so?
I'm running on Linux RedHat 5.2
A ...
3
votes
0
answers
517
views
rsync won't delete file that was deleted on source
I'm using rsnapshot to back up some servers of mine. It works fine, except it won't delete files that have been deleted on the source server. Here is where I give rsnapshot the options to pass along ...
- 317
3
votes
1
answer
714
views
Rackspace cloud sites to local server backup
I use rackspace cloud sites and don't have accsess to SSH or RSYNC on their servers, but I do access to SSHFS.
I would like to be able to backup my cloud site to my local Ubuntu server, which has SSH,...
- 137
2
votes
0
answers
1k
views
Why on earth would rsync over SMB run nearly five times faster than rsync-to-rsync (over SSH)?
I just modified many files on a 36TB Unraid array (not quite half full yet). Now I want to sync all of those changes up to a backup array using rsync.
My usual rsync backup was going a lot more slowly ...
- 129
2
votes
2
answers
211
views
rsync uncompressed to single compressed archive
Question
How can I rsync uncompressed (but highly compressible) data to a single compressed archive (.tar.gz, .zip, etc) without transferring all the data each time?
Current
This is what I am using ...
- 121
2
votes
0
answers
956
views
ansible.posix.synchronize fails with unknown rsync option
I am running an ansible playbook with the following task:
- name: Yada Yada Yada
ansible.posix.synchronize:
src: ../foo/bar/
dest: /home/tada/tokio
When i run this using AWX i get the ...
- 121
2
votes
2
answers
547
views
Moving files with rsync and custom port results in permission denied
I'm trying to move some files between servers with rsync and non-standard SSH port. I can login when I just use ssh and supply my password. When I try with rsync I get permission denied. I'm not sure ...
- 121
2
votes
0
answers
174
views
Use local ssh_config settings on a remote server
Good day, friends!
I've got a locally configured ~/ssh/config file with all the hosts set up as they should. I am using them successfully when doing my ssh from the local computer like ssh dev. But ...
- 141
2
votes
1
answer
1k
views
Fastest way to tranfer data over the network between two servers
I have two storage servers at home - one is my workstation, which has a pair of 3TB disks in RAID1.
At first, I stored all of my data there. Recently, I received a small SBC with four 1TB disks.
I set ...
- 165
2
votes
0
answers
3k
views
Is there a way to tell what bottle neck is in slow rsync transfer
I am transferring a directory from a USB hard drive to my NAS device (SMB share).
USB Hard Drive performance:
sudo hdparm -Tt /dev/sdb
/dev/sdb:
Timing cached reads: 27884 MB in 1.99 seconds = ...
- 285
2
votes
0
answers
1k
views
(How) should I lock a Maildir mailbox managed by Dovecot before creating a backup via rsync?
I maintain an IMAP server running Dovecot which uses Maildir as the storage format for the user mailboxes. Every night, an rsync process runs to create a backup of the mailboxes. Recently, I noticed ...
- 791
2
votes
0
answers
10k
views
How to kill rsync daemon that keeps respawning
How do I kill these rsync daemons that keep respawning
root@que2 ~ # ps aux | grep rsync
root 9660 0.0 0.0 19928 2844 ? S 13:35 0:00 rsync -rlptD --progress --delete --exclude=server/...
- 337
2
votes
0
answers
231
views
Mitigate rsync's burstiness
We use rsync as backup on a NFS share we have on our VPN network.
The ISP's modem is awful and it stops responding when we ask too much upload bandwidth to it. We have 150Mbps down / 20 Mbps up ...
- 155
2
votes
1
answer
1k
views
Why is rsnapshot with sync_first not working properly?
I had been using rsnapshot successfully for years to backup my Ubuntu notebook computers to my Ubuntu server. I recently decided to switch to using the sync_first option on the server and an anacron ...
- 21
2
votes
0
answers
2k
views
How to reduce the IO Wait Time during a rsync backup?
I have the following situation: There is a set of machines in the network (NAS and other "servers"). There is one additional machine for backup purposes. It collects the data from all 4 machines on a ...
- 308
2
votes
0
answers
2k
views
rsync error after a while: "rsync service is no running"
I am migrating data from an old WD NAS to a Synology NAS via rsync. For some reason, my rsync processes are killed after a while, with the error message:
rsync error: rsync service is no running (...
- 441
2
votes
0
answers
852
views
Delete my unison folder cant find how to sync again all
I run unison sync and it works perfect, but accidentally dropped the folder and now I cannot recover it. I tried to run unison again, but it returned
Nothing to do: replicas have not changed since ...
- 21
2
votes
1
answer
743
views
What is limiting bandwidth using rsync
we are running a 10Gbit server. Testing whith iperf works perfect:
[ 3] 0.0-10.0 sec 10.6 GBytes 9.15 Gbits/sec
But when using rsync (with ssh on another port: rsync -Pe 'ssh -p xxx') the ...
- 439
2
votes
0
answers
529
views
Rsync transfer-log on server site?
I have a backup script that copies contents of user directories on local workstations using rsync to another server. I need to use ssh for that because I don't want to run a rsyncd on the backup ...
- 237
2
votes
0
answers
380
views
rsync: Maximum nice to keep system responsive
Unfortunately a background rsync reduces the responsiveness of a server.
I search a way to have maximum niceness for this background task, to keep the server respond fast.
My current wrapper ...
- 3,843
2
votes
0
answers
104
views
keep writing to file after network filesystem disconnects
I'm trying to transfer a huge growing logfile in a way that will survive network partitions between the client and server. If the network connection is lost, I cannot afford to retransmit the entire ...
- 121
2
votes
0
answers
282
views
rsync ionice softraid - limit io locally?
I know this seems to be a recurrent question but it's been troubling for a while now, so if anyone has a brilliant idea please let me know.
My Setup
OS
Debian Wheezy
HDDs
2 x 240GB SSD for system ...
- 73
2
votes
0
answers
3k
views
Simple progress for rsync when piping output to a log file
I ran rsync with nohup and --progress:
nohup rsync --progress --[other-options] source destination &
tail -f nohup.out
When I'm tailing the file, I see the output with a changing last line (...
- 153
2
votes
0
answers
375
views
How do I prevent rsync failure from breaking my ability to SSH into remote AWS server?
I'm running into this problem where if my rsync to an Amazon CentOS instance breaks mid-transfer (with a Ctrl+C for example), subsequent logins to the server no longer works. The Server just responds ...
- 21
2
votes
2
answers
2k
views
How to increase the disk write buffer for rsync on Mac OS X?
I have to copy some TB of data from one external drive to another (mostly ~400 MB files).
The source is NTFS-formatted USB2 and the best read speed I can get from it is 7MB/s.
The target is HFS+ USB3 ...
- 185
2
votes
0
answers
2k
views
What is a fast, efficient method for threading rsync
I have a single root directory (on a NAS) that I need to backup to another server. The hierarchy of the directory is several layers deep and a majority of the data is composed of extremely large ...
- 270
2
votes
0
answers
411
views
Backup with local manifest
I use rsync to backup files to a remote server in a UNIX environment. As my local machine is the only one with write access to the backup files, I know the status of the backup files on the remote ...
- 121
2
votes
1
answer
722
views
rsnapshot Freezes when Backing up to NFS Mount
I've got a Server running Ubuntu Server 14.04 server that has been doing periodic back-ups to an NFS mounted NAS using rsnapshot for several months now without issue. Last week we added a few more ...
- 119
2
votes
0
answers
877
views
rsync daemon without rsyncd.conf
Will rsyncd run without its config file (rsyncd.conf) ? What is its behavior if run this way?
Backstory:
So, I'm using rsync to backup user directories from (call it) server A, and it's working ...
- 615
2
votes
1
answer
1k
views
rsync using only spare bandwidth?
My scenario:
Several rural schools
Every school:
One linux server
Limited bandwidth (shared with all school PCs)
Electricity not guaranteed (last to leave the school could cut all electricity)
...
- 21
2
votes
0
answers
82
views
List subfolders somewhere in the path with rsync
I'm developing a program, that will recursively "walk" through Linux repositories, look for PXE-related files and create gPXE configs.
I met across a problem with the structure of Ubuntu catalogs on ...
- 133
2
votes
2
answers
995
views
What's the best way to Deploy/Update/Control Windows services on a hundred EC2 servers?
I'm developing an image processing application running on Windows on Amazon EC2. The application is currently a console app I'm starting manually on a few machines. I'm refactoring it into a Windows ...
- 143
2
votes
0
answers
3k
views
rsyncd.conf exclude from configuration not working
I'm trying to use a text file to contain the excluded files for the rsyncd daemon. I followed the rsyncd.con manual and put:
exclude from = /etc/rsync-exclude.txt
But it seems not working. What ...
- 217
2
votes
0
answers
319
views
Syncrify and empty folders
Perhaps I'm missing something in the configuration, but can anyone tell me how to force Syncrify backup empty folders?
It seems if they're empty they don't get created in the backup.
- 7,907
1
vote
1
answer
70
views
extract full path FIND with special character handling
Executing a comand like:
find /some_path -mtime -3 -type f -name "*.txt"
the out is something like:
/some_path/file 1.txt
/some_path/file2.txt
If I include this output in rsync with ...
- 11
1
vote
0
answers
98
views
Backup drive on Linux system keeps getting corrupt. How to troubleshoot?
My webserver (Debian BullsEye) has an internal 2TB harddrive which sole purpose is to hold a backup of the server's main HD. Every day a cronjob is started that creates a backup using rsync. But every ...
- 279
1
vote
0
answers
351
views
Samba AD sysvol sync unison permissions issue
I'm having an issue with syncing the unix permissions and ownership on the sysvol between two Ubuntu Samba AD DCs using unison. I have copied the idmap.ldb file from DC1 to DC2 and the Windows ...
1
vote
0
answers
293
views
How suitable is a S3 backend for rsync operations?
I run a large server providing open source software (https://ftp.halifax.rwth-aachen.de), currently serving more than 30 TByte of data with Multi-Gigabit throughput. Data is synchronized and kept up-...
- 374
1
vote
0
answers
679
views
Kernel files (linux-image) to exclude on rsync full clone to different hardware
EDIT
Now I'm able to boot the OS.
As I made clean copy of the /etc directory just after buying the "new" dedicated server and installing Ubuntu server 20.04, i proceeded to:
Boot the "...
- 508
1
vote
0
answers
298
views
Rsync: Exclude files by maximum filename length (< 143 characters)
A cronjob copies files from a remote server (probably ext4) via rsync to a local encrypted partition (probably eCryptFS on top of ext4).
Now rsync throws errors for filenames that are longer than 143, ...
- 143