Hot answers tagged

57

I have had very good results using tar, pigz (parallel gzip) and nc. Source machine: tar -cf - -C /path/of/small/files . | pigz | nc -l 9876 Destination machine: To extract: nc source_machine_ip 9876 | pigz -d | tar -xf - -C /put/stuff/here To keep archive: nc source_machine_ip 9876 > smallstuff.tar.gz If you want to see the transfer rate just ...


20

I'd stick to the rsync solution. Modern (3.0.0+) rsync uses incremental file list, so it does not have to build full list before transfer. So restarting it won't require you to do whole transfer again in case of trouble. Splitting the transfer per top or second level directory will optimize this even further. (I'd use rsync -a -P and add --compress if your ...


15

Set up a VPN (if its internet), create a virtual drive of some format on the remote server (make it ext4), mount it on the remote server, then mount that on the local server (using a block-level protocol like iSCSI), and use dd or another block-level tool to do the transfer. You can then copy the files off the virtual drive to the real (XFS) drive at your ...


9

If the old server is being decommissioned and the files can be offline for a few minutes then it is often fastest to just pull the drives out the old box and cable them into the new server, mount them (back online now) and copy the files to the new servers native disks.


5

Quoting man ssh (which is the base used by scp): Compression is desirable on modem lines and other slow connections, but will only slow down things on fast networks. The problem is that compressing the data takes more time then just sending it over the network.


5

In the general case, the HDD will be the limiting factor for how fast that operation will go. In the specific case of doing a block copy from the HDD to the SSD, which is a 100% sequential operation, the HDD will keep up. If it is a file-copy, there will be enough jitter in the disk-access pattern that it will lag the SSD enough to be the limiting factor; ...


3

Have you considered sneakernet? With that, I mean transfering everything onto the same drive, then physically moving that drive over. about a month ago, Samsung unveiled a 16 TB drive (technically, it's 15.36 TB), which is also an SSD: http://www.theverge.com/2015/8/14/9153083/samsung-worlds-largest-hard-drive-16tb I think this drive would just about do ...


3

Use mbuffer and if it is on a secure network you can avoid the encryption step.


2

The only way you'll be able to checksum an entire drive is to un-mount the desired filesystem and then checksum it: (assuming that /dev/sdb is mounted to /mnt/foo) $ umount /mnt/foo $ md5sum /dev/sdb ... Once you have the checksum, you can use dd piped through ssh to transfer the entire block device to another system: $ dd if=/dev/sdb | ssh user@host ...


2

Something like: ssh user@serverB nohup wget -bqc ftp://path/largefile.tar.gz wget options: -b : run in background -q : quiet -c : resume broken download (means you can restart if it breaks) This runs wget in the background so (hopefully) if you exit the ssh shell it'll keep going. Ok, I think you need nohup to ensure that is the case when/if you ...


2

Try inserting a space before the destination, like this: robocopy "c:\transfer_this.txt" "z: \this.txt" notice the space after the destination "folder" z:.


2

(Many different answers can work. Here is another one.) Generate the file list with find -type f (this should finish in a couple of hours), split it to small chunks, and transfer each chunk using rsync --files-from=....


2

You are using RedHat Linux, so this wouldn't apply, but as another option: I've had great success using ZFS to hold millions of files as inodes aren't an issue. If that was an option for you, you could then take snapshots and use zfs to send incremental updates. I've had a lot of success using this method to transfer as well as archive data. ZFS is ...


2

If there is any chance to get high success ratio when deduplication, I would use something like borgbackup or Attic. If not, check the netcat+tar+pbzip2 solution, adapt the compression options according to your hardware - check what is the bottleneck (CPU? network? IO?). The pbzip2 would nicely span across all CPUs, giving better performance.


2

For a first look, you can open the task manager and see if the DISK column shows high load. For a more detailed investigation, you had to use the Performance Monitor application. Follow these steps: open it under "performance monitor", click add (+ sign) and monitor these two counters: Cache/dirty pages: the amount of the write cache, in 4KB pages ...


1

Using tmux/Screen would be a more preferable way instead of nohup. You can always reattach the terminal, in case you loose the connection. For the file transfer itself I would recommend using SSH+Rsync. Rsync can resume files transfers, and it will be encrypted too. Try something like: rsync -av --partial server1:/my/dir server2:/this/dir


1

Start an rsync daemon on the target machine. This will speedup the transfer process a lot.


1

You can also try using the BBCP command to do your transfer. It's a buffered parallel ssh that really screams. We can usually get 90%+ line-rate provided we can keep the pipe fed. $ bbcp -s 8 -w 64M -N io 'tar -cO srcdirectory' desthostname:'tar -x -C destdir' Normally, we try real hard to avoid having to move suff around. We use ZFS pools that we ...


1

If it's not very sensitive data and your connection is safe enough, ssh into B and download straight from A via ftp. SSH will make your download considerably slower because of the encryption work overhead. If possible split the 100GB file in multiple ones, especially if the ftp server on A doesn't support download resume.



Only top voted, non community-wiki answers of a minimum length are eligible