183
votes
Accepted
Rsync triggered Linux OOM killer on a single 50 GB file
So let us read the oom-killer output and see what can be learned from there.
When analyzing OOM killer logs, it is important to look at what triggered it. The first line of your log gives us some ...
- 40.5k
94
votes
Can I make rsync output only the summary?
Thanks to a tip by Wayne Davison, I use the --stats option for backup:
rsync -a --stats src/ dest/
Nice little summary at the end, e.g.
Number of files: 6765
Number of files transferred: 0
Total file ...
- 1,042
55
votes
rsync files to a kubernetes pod
To rsync to a Pod I use the following helper.
pod=$1;shift;kubectl exec -i $pod -- "$@"
I put this in a file called "rsync-helper.sh" and then run the rsync like so.
rsync -av --...
- 651
54
votes
Accepted
Rsync to AWS S3 bucket
To communicate to s3 you need to have 2 things
IAM user credentials who has read-write access to s3 bucket.
A client like aws-cli for bash, boto library for Python etc.
once you have both, you can ...
- 1,396
48
votes
rsync: getcwd(): No such file or directory (2)
I had the exact same issue, and my solution was something I had never thought of. In my case rsync worked fine for all the time until it stopped during some tests after changes to one of my scripts. ...
45
votes
Using rsync to only delete extraneous files
You need --existing too:
rsync -r --delete --existing --ignore-existing /home/gallery/images /home/gallery/thumbs
From the manpage:
--existing, --ignore-non-existing
This tells rsync to ...
- 1,562
44
votes
Accepted
Faster rsync of huge directory which was not changed
Some unrelated points:
80K is a lot of files.
80,000 files in one directory? No operating system or app handles that situation very well by default. You just happen to notice this problem with rsync....
- 7,767
38
votes
Using rsync from msysgit for binary files
As of now (9/7/2017) it is extremely easy to manually add rsync support to the Git for Windows environment without even having to resort to running the Git for Windows SDK.
I stumbled across this ...
- 821
35
votes
Accepted
How do I set destination permissions with rsync (chown chmod)
rsync needs to be told that you want to set the permissions and owner/group information. It would be logical to assume that having --chmod or --chown would tell that but they don't.
For permissions ...
- 466
31
votes
Rsync to AWS S3 bucket
You can try rclone.
Setup (docs):
rclone config
Sync (docs):
rclone sync /my/local/folder s3service:bucket-name
- 1,215
28
votes
Is it possible to use rsync over sftp (without an ssh shell)?
An alternative to using rsync is to instead use lftp (which can connect to sftp) and use the mirror command. For example one can do
lftp
~> open -u user,password sftp://host.com
~> mirror ...
- 549
28
votes
Perform rsync while following sym links
You need both -L and -K if you have symlinks on both sides, e.g. you already had your 1st rsync done and want to update the backup using rsync.
-L, --copy-links transform symlink into ...
- 381
22
votes
Perform rsync while following sym links
Just ran into this problem. And if you want rsync to treat symlinked directories as directories, you want the K option
rsync -K /files/ user@server:/files/
- 341
22
votes
Run totally silent rsync?
quiet option should do that:
-q, --quiet suppress non-error messages
- 1,289
22
votes
Accepted
rsync - exclude all directories except a few
A simple filter should do the trick. To build on the prior answer with a proper example -- Explicitly include the parent(s), plus all (**) sub folders and files. Then exclude everything else. Here's ...
- 421
20
votes
Accepted
rsync --compress-level: which compression levels can be used?
It's values between 0-9. Where 1 is fastest and 9 is the most compressed.
Other than that there's correlation between rsync and zlib where
rsync tells the zlib library to "use the default ...
- 424
19
votes
How to keep the full path with rsync?
Another common related use case it to keep a selected part of the directory tree with /./, e.g. if you wanted:
/home/udi/files/pictures
to go to:
backup/files/pictures
you can write:
rsync --...
19
votes
Restricting a ssh key to only allow rsync/file transfer?
rrsync is designed to be used as a forced command for a particular key, so it should be exactly what you want.
A forced command is set up using the command option for a key in an authorized keys file ...
16
votes
Accepted
Preserve ownership with rsync, without root
As stated in the other answers, to directly preserve ownership information you need root access to the destination machine.
However, you had at least two workarounds to avoid root access while ...
- 45.5k
16
votes
Accepted
rsync error: error in rsync protocol data stream (code 12) at io.c(600) [sender=3.0.6]
rsync is not installed on 999.999.99.9
- 1,554
15
votes
Copying a large directory tree locally? cp or rsync?
This thread was very useful and because there were so many options to achieve the result, I decided to benchmark few of them. I believe my results can be helpful to others have a sense of what worked ...
- 251
15
votes
Rsync daemon: is it really useful?
rsyncd (rsyncd with daemon) uses by default port 873.
This will use the native rsync protocol, but does not encrypt anything. It is meant only for local or trusted networks.
If you want security, ...
- 151
15
votes
Can rsync display current average speed?
In case you're not running v3.1.0, rsync-3.0.9 takes the --progress flag. The man page on my CentOS7 box reports
This option tells rsync to print information showing the progress of the transfer. ...
- 345
15
votes
rsync : write failed - No space left on device (28)
I had the same issue, the destination directory had enough space but I'd get "No space left on device". Turns out rsync copies the file to somewhere else first, then moves it to the destination ...
- 251
15
votes
Accepted
Does rsync change file content (dealing with hardlinks)?
With standard options, rsync will copy the new file assigning it a semi-random name, then it will rename the new file with the original name. In this process, no writes are directed at the original ...
- 45.5k
14
votes
How to delete empty source directories when moving folders with rsync?
I ended up in a similar situation. I wanted to avoid rm -rf, knowing that rsync failed to copy some files, and I didn't want to loose them.
To simply delete the empty directories, I found this one ...
- 241
14
votes
How to tell rsync do not check permissions
If you supply the -a flag -p is automatically included. If you specify --no-p all of the other options from -a remain intact.
- 241
14
votes
rsync output file sizes in human-readable format
Reading the --human-readable section of the man page reveals
-h, --human-readable
Output numbers in a more human-readable format. This makes big numbers output using larger units, with a K, M,...
- 115k
13
votes
Accepted
How do "correctly" sync millions of files with rsync and inotify?
Your case sounds exactly what lsyncd was made for.
It watches the directory with inotify and spawns an rsync with only the changed file(s).
It aggregates changes for a few seconds.
So when you make 5 ...
- 17.4k
12
votes
rsync - report only uploaded files
rsync with a single -v|--verbose actually prints only transferred files (as well as a header/footer). If you are getting the complete list every time then it probably means that the default rsync src/...
- 353
Only top scored, non community-wiki answers of a minimum length are eligible