Skip to main content
24 votes

Easy way to transfer files between host and LXC container on LVM

I know that this is an old question, but for someone ending up here while searching for how to copy files between host and container, this might help. To pull a file 'my-file' from the container '...
Ludwig's user avatar
  • 441
17 votes

Copying a large directory tree locally? cp or rsync?

This thread was very useful and because there were so many options to achieve the result, I decided to benchmark few of them. I believe my results can be helpful to others have a sense of what worked ...
arjones's user avatar
  • 271
11 votes

Get list of transferred files from rsync?

Use the --out-format option According to the man page: Specifying the --out-format option will mention each file, dir, etc. that gets updated in a significant way (a transferred file, a ...
flu's user avatar
  • 211
9 votes
Accepted

How do increase data transfer rate between my VMs

Since it seems to be a solution, I am adding it as answer: When creating a VM, the default network adapter is an emulated Intel E1000E. This adapter works in most operating systems without additional ...
Tobias's user avatar
  • 1,246
8 votes

Storage Space Write cache, and write speed

This postponed write happens because of a a) file system level write-back RAM cache, and b) parity Spaces design, which absorbs all the writes to the internal log initially to avoid “read-modify-write”...
BaronSamedi1958's user avatar
6 votes

Easy way to transfer files between host and LXC container on LVM

A better, somehow built-in way, of transfering data from host to lxc-container is line #4 in below code: 1 $ mkdir /tmp/transferDir 2 $ cp <some files> /tmp/transferDir/<some files> 3 $ ...
flipcc's user avatar
  • 61
6 votes

Copying terabytes of hundreds of thousands of files in folder slow

The root cause for the slow transfer rate is, possibly, the fact that the workstation M2 drives need to do a lot of random reads. The fast NVMe M2s (that you are most likely using, I think) are ...
Jules's user avatar
  • 61
5 votes

Is there an equivalent to ssh-copy-id for Windows?

If you do not have ssh-copy-id on Windows, you can run it on the server itself. In PuTTYgen, load your private key (.ppk); Copy the contents of the box Public key for pasting into OpenSSH ...
Martin Prikryl's user avatar
5 votes

Server OS or PowerShell version differences that would affect copy speed?

(1) DAS -> RAM -> Network -> SAN Vs. (2) SAN -> Network -> RAM -> Network -> SAN Two network hops instead of just one for every single I/O within your second scenario. Network is slow compared to ...
BaronSamedi1958's user avatar
4 votes

Copying a large directory tree locally? cp or rsync?

There are some speed-ups which can be applied to rsync: Avoid -z/--compress: compression will only load up the CPU as the transfer isn't over a network but over RAM. --append-verify: resume an ...
Tom Hale's user avatar
  • 1,165
4 votes

Copying a large directory tree locally? cp or rsync?

You definitely want to give rclone a try. This thing is crazy fast : sudo rclone sync /usr /home/fred/temp -P -L --transfers 64 Transferred: 17.929G / 17.929 GBytes, 100%, 165.692 MBytes/s, ...
Frédéric N.'s user avatar
4 votes
Accepted

What is the most efficient way to transfer files from AWS S3 to S3?

https://aws.amazon.com/premiumsupport/knowledge-center/account-transfer-s3/ You can copy Amazon S3 objects from one AWS account to another by using the S3 COPY operation. You must give the ...
ceejayoz's user avatar
  • 33.2k
4 votes
Accepted

copy large number of files over ssh

…to answer the original question as stated… There are two things to discuss here. Using SSHFS SSHFS uses the SFTP "subsystem" of the SSH protocol to make a remote filesystem appear as if it were ...
kostix's user avatar
  • 1,160
4 votes
Accepted

How to block Filetransfer through RDP (Port 3389)?

I assume you are asking how to block copy/paste files from local PC to remote desktop (Windows 10) through RDP window directly. If yes, please config the group policy in remote desktop server (Windows ...
yanqian's user avatar
  • 141
3 votes

Secure file transfer in linux without SSH?

The following is simple and awesome: python -m SimpleHTTPServer 8888 This starts a new http file server for current directory. Supposed there is a file named foo in current directory, you can ...
nullne co's user avatar
3 votes

Get list of transferred files from rsync?

Method 1 You can use --info=NAME option(only versions later than rsync 3.1.0 are supported) rsync -azPrui --info=NAME /path/from/source/ /path/to/destination/ use rsync --info=help to get the value ...
Willis's user avatar
  • 131
3 votes
Accepted

HTTP to FTP with remote client

Mount the ftp directory: curlftpfs ftp.yourserver.com /mnt/ftp/ -o user=username:password Then you can easily download the selected files to this folder: wget http://user:[email protected]/somefile -...
Ipor Sircer's user avatar
  • 1,236
3 votes

Why is My Network file Transfer Slower Than (What I Thought) it Should Be?

150Mbps would translate to 8.9MB/s. Remember, all the data has to be sent twice, once to the access point and once from the access point. But that would include no address and control overhead, a ...
David Schwartz's user avatar
3 votes
Accepted

Copying terabytes of hundreds of thousands of files in folder slow

Ok looks like the issue is resolved now. Here was the solution. In the /etc/samba/smb-shares.conf.local This line was added to the share we are using case sensitive = yes Now we are transferring ...
cohortq's user avatar
  • 475
3 votes
Accepted

Copy large amounts of data from S3 Bucket to Bucket, different regions

If you're copying data that is already in AWS to another region, Snowball doesn't seem like a good option. Running sync from the command line of an EC2 instance within AWS seems right. Have you ...
vjones's user avatar
  • 871
3 votes

Best way to transfer about 200GB of files between two windows servers

Robocopy is good for moving a lot of data. Command help here - https://ss64.com/nt/robocopy.html
Edwrd_T_Justice's user avatar
3 votes

Best way to transfer about 200GB of files between two windows servers

Is downtime acceptable at all? Sneakernet is still a thing, and depending on the physical distance it may provide enormous bandwidth. For machines with limited allowed downtime it might still be an ...
Mikael H's user avatar
  • 5,129
2 votes

How to copy a large number of files quickly between two servers

Thanks to Scott Pack's wonderful answer (I didn't know how to do this with ssh before), I can offer this improvement (if bash is your shell). This will add parallel compression, a progress indicator ...
Daniel Santos's user avatar
2 votes

Which FTP server on Windows or Linux do support MODE C (Compression)

I will take the time to answer this question better, although, in essence, Aaron is right. There are no FTP servers on Windows, Linux or UNIX that support MODE C. The z/OS FTP server only supports ...
thecarpy's user avatar
  • 176
2 votes

Multicast File Transfers

uftp is working for me. I ended up maxing the "robust" -s option to 50, and opening the server firewall to the ip's of the clients for the registration phase.
js2010's user avatar
  • 183
2 votes

Checksumming to verify rsync transfers

Use rsync -Pahn --checksum /path/to/source /path/to/destination | sed '/\/$/d' | tee migration.txt sed removes directories from the checksum verification. tee outputs to the screen and to the file at ...
Gaia's user avatar
  • 1,925
2 votes

Rsync hangs when transfering large files on an bad connection

As we cannot know the cause of connection problems, I can suggest the --partial option of rsync. It simply is able to restart an interrupted download/upload. --partial By default, rsync will ...
Echoes_86's user avatar
  • 173
2 votes
Accepted

Debug slow lan (ssh, nfs) file transfer

At the numbers you get, seems like something is not actually gigabit in your network. Use ethtool or something similar to check the speed on each machine NIC. Check the switch and make sure the ...
Overmind's user avatar
  • 3,171
2 votes
Accepted

Will the update flag in rsync transfer files if the timestamp is identical in source and destination? Documentation is unclear

Found the following in the man expanding on the flag: -u, --update This forces rsync to skip any files which exist on the destination and have a modified time that is newer than the source file. (If ...
André Christoffer Andersen's user avatar
2 votes

Transfer 300+ GB at maximum speed on two headless servers?

Are you sure the limit isn't from the HDDs? SSH with a faster cipher can help: ssh -c arcfour scp -c arcfour Or you could run rsync without sshd, with rsyncd. Another option is to pipe tar over ...
ptman's user avatar
  • 29.2k

Only top scored, non community-wiki answers of a minimum length are eligible