66
votes
Accepted
Transfer 15TB of tiny files
I have had very good results using tar, pigz (parallel gzip) and nc.
Source machine:
tar -cf - -C /path/of/small/files . | pigz | nc -l 9876
Destination machine:
To extract:
nc source_machine_ip ...
- 2,756
21
votes
Easy way to transfer files between host and LXC container on LVM
I know that this is an old question, but for someone ending up here while searching for how to copy files between host and container, this might help.
To pull a file 'my-file' from the container '...
- 411
21
votes
Transfer 15TB of tiny files
I'd stick to the rsync solution. Modern (3.0.0+) rsync uses incremental file list, so it does not have to build full list before transfer. So restarting it won't require you to do whole transfer again ...
- 3,927
15
votes
Copying a large directory tree locally? cp or rsync?
This thread was very useful and because there were so many options to achieve the result, I decided to benchmark few of them. I believe my results can be helpful to others have a sense of what worked ...
- 251
15
votes
Transfer 15TB of tiny files
Set up a VPN (if its internet), create a virtual drive of some format on the remote server (make it ext4), mount it on the remote server, then mount that on the local server (using a block-level ...
- 471
13
votes
Accepted
Why is scp with compression slower than without?
Quoting man ssh (which is the base used by scp):
Compression is desirable on modem lines and other slow connections, but will only slow down things on fast networks.
The problem is that ...
- 98k
12
votes
Accepted
Easy way to transfer files between host and LXC container on LVM
Revised answer: LXC containers share the same kernel as the host, so any filesystem they mount should be accessible from outside.
If you do a cat /proc/mounts on the host, can you see the container ...
- 1,173
10
votes
Get list of transferred files from rsync?
Use the --out-format option
According to the man page:
Specifying the --out-format option will mention each file, dir, etc.
that gets updated in a significant way (a transferred file, a
...
- 201
10
votes
Transfer large amount of small files
If you really need a quick way to transfer files, and both systems are Linux-based, you can try UDR.
This is really a form of rsync-over-UDP (using the open-source UDT framework) and is particularly ...
- 196k
10
votes
Transfer 15TB of tiny files
If the old server is being decommissioned and the files can be offline for a few minutes then it is often fastest to just pull the drives out the old box and cable them into the new server, mount them ...
- 121
9
votes
Get list of transferred files from rsync?
Beginning with rsync v3.1.0, released in 2013, there is the --info flag which allows fine-grained control over the output.
--info=FLAGS
This option lets you have fine-grained control over ...
- 203
9
votes
Accepted
How do increase data transfer rate between my VMs
Since it seems to be a solution, I am adding it as answer:
When creating a VM, the default network adapter is an emulated Intel E1000E. This adapter works in most operating systems without additional ...
- 1,236
8
votes
Storage Space Write cache, and write speed
This postponed write happens because of a a) file system level write-back RAM cache, and b) parity Spaces design, which absorbs all the writes to the internal log initially to avoid “read-modify-write”...
- 12.7k
6
votes
Easy way to transfer files between host and LXC container on LVM
A better, somehow built-in way, of transfering data from host to lxc-container is line #4 in below code:
1 $ mkdir /tmp/transferDir
2 $ cp <some files> /tmp/transferDir/<some files>
3 $ ...
- 61
6
votes
Accepted
Are HDD <-> SSD transfer speeds completely bottlenecked by HDD speed?
In the general case, the HDD will be the limiting factor for how fast that operation will go. In the specific case of doing a block copy from the HDD to the SSD, which is a 100% sequential operation, ...
- 132k
6
votes
Copying terabytes of hundreds of thousands of files in folder slow
The root cause for the slow transfer rate is, possibly, the fact that the workstation M2 drives need to do a lot of random reads.
The fast NVMe M2s (that you are most likely using, I think) are ...
- 61
5
votes
Is there an equivalent to ssh-copy-id for Windows?
If you do not have ssh-copy-id on Windows, you can run it on the server itself.
In PuTTYgen, load your private key (.ppk);
Copy the contents of the box Public key for pasting into OpenSSH ...
- 7,524
5
votes
Is there an equivalent to ssh-copy-id for Windows?
In Windows 7 there is a ssh.exe
Here is what worked for me:
1. create identity (on windows)
c:\>ssh-keygen
That created an identity file in the home directory.
I changed the name of the public ...
- 151
5
votes
Server OS or PowerShell version differences that would affect copy speed?
(1) DAS -> RAM -> Network -> SAN
Vs.
(2) SAN -> Network -> RAM -> Network -> SAN
Two network hops instead of just one for every single I/O within your second scenario. Network is slow compared to ...
- 12.7k
4
votes
Copying a large directory tree locally? cp or rsync?
You definitely want to give rclone a try. This thing is crazy fast :
sudo rclone sync /usr /home/fred/temp -P -L --transfers 64
Transferred: 17.929G / 17.929 GBytes, 100%, 165.692 MBytes/s, ...
- 49
4
votes
Openssh sftp-server: .filepart support?
The OpenSSH server does not support this.
WinSCP SFTP client can do this from a client side, if that helps.
See https://winscp.net/eng/docs/resume#automatic
By default is uses the .filepart suffix ...
- 7,524
4
votes
Accepted
What is the most efficient way to transfer files from AWS S3 to S3?
https://aws.amazon.com/premiumsupport/knowledge-center/account-transfer-s3/
You can copy Amazon S3 objects from one AWS account to another by using the S3 COPY operation. You must give the ...
- 32.7k
4
votes
Accepted
copy large number of files over ssh
…to answer the original question as stated…
There are two things to discuss here.
Using SSHFS
SSHFS uses the SFTP "subsystem" of the SSH protocol
to make a remote filesystem appear as if it were ...
- 1,120
4
votes
Accepted
How to block Filetransfer through RDP (Port 3389)?
I assume you are asking how to block copy/paste files from local PC to remote desktop (Windows 10) through RDP window directly.
If yes, please config the group policy in remote desktop server (Windows ...
- 141
3
votes
Secure file transfer in linux without SSH?
The following is simple and awesome:
python -m SimpleHTTPServer 8888
This starts a new http file server for current directory. Supposed there is a file named foo in current directory, you can ...
- 49
3
votes
Transfer large amount of small files
If used in daemon mode without encryption, rsync can efficiently transfer large amount of small files. Give it another try using it in daemon mode.
- 45.4k
3
votes
Transfer 15TB of tiny files
Use mbuffer and if it is on a secure network you can avoid the encryption step.
- 8,158
3
votes
Transfer 15TB of tiny files
(Many different answers can work. Here is another one.)
Generate the file list with find -type f (this should finish in a couple of hours), split it to small chunks, and transfer each chunk using ...
- 435
3
votes
Transfer 15TB of tiny files
Have you considered sneakernet? With that, I mean transfering everything onto the same drive, then physically moving that drive over.
about a month ago, Samsung unveiled a 16 TB drive (technically, ...
- 331
3
votes
Accepted
HTTP to FTP with remote client
Mount the ftp directory:
curlftpfs ftp.yourserver.com /mnt/ftp/ -o user=username:password
Then you can easily download the selected files to this folder:
wget http://user:pass@example.org/somefile -...
- 1,226
Only top scored, non community-wiki answers of a minimum length are eligible