Hot answers tagged

56

I have had very good results using tar, pigz (parallel gzip) and nc. Source machine: tar -cf - -C /path/of/small/files . | pigz | nc -l 9876 Destination machine: To extract: nc source_machine_ip 9876 | pigz -d | tar -xf - -C /put/stuff/here To keep archive: nc source_machine_ip 9876 > smallstuff.tar.gz If you want to see the transfer rate just ...


20

I'd stick to the rsync solution. Modern (3.0.0+) rsync uses incremental file list, so it does not have to build full list before transfer. So restarting it won't require you to do whole transfer again in case of trouble. Splitting the transfer per top or second level directory will optimize this even further. (I'd use rsync -a -P and add --compress if your ...


15

Set up a VPN (if its internet), create a virtual drive of some format on the remote server (make it ext4), mount it on the remote server, then mount that on the local server (using a block-level protocol like iSCSI), and use dd or another block-level tool to do the transfer. You can then copy the files off the virtual drive to the real (XFS) drive at your ...


10

If you really need a quick way to transfer files, and both systems are Linux-based, you can try UDR. This is really a form of rsync-over-UDP (using the open-source UDT framework) and is particularly handy for moving large numbers of files or transferring over high-bandwidth or high-latency links. In addition, encryption is disabled by default, so the ...


9

If the old server is being decommissioned and the files can be offline for a few minutes then it is often fastest to just pull the drives out the old box and cable them into the new server, mount them (back online now) and copy the files to the new servers native disks.


5

Quoting man ssh (which is the base used by scp): Compression is desirable on modem lines and other slow connections, but will only slow down things on fast networks. The problem is that compressing the data takes more time then just sending it over the network.


5

In the general case, the HDD will be the limiting factor for how fast that operation will go. In the specific case of doing a block copy from the HDD to the SSD, which is a 100% sequential operation, the HDD will keep up. If it is a file-copy, there will be enough jitter in the disk-access pattern that it will lag the SSD enough to be the limiting factor; ...


3

Have you considered sneakernet? With that, I mean transfering everything onto the same drive, then physically moving that drive over. about a month ago, Samsung unveiled a 16 TB drive (technically, it's 15.36 TB), which is also an SSD: http://www.theverge.com/2015/8/14/9153083/samsung-worlds-largest-hard-drive-16tb I think this drive would just about do ...


3

Revised answer: LXC containers share the same kernel as the host, so any filesystem they mount should be accessible from outside. If you do a cat /proc/mounts on the host, can you see the container filesystems? If you see a line like /dev/mapper/... /var/lib/lxc/o1/rootfs ext4 ... then you should be able to access /var/lib/lxc/o1/rootfs from the host, ...


3

If used in daemon mode without encryption, rsync can efficiently transfer large amount of small files. Give it another try using it in daemon mode.


3

Use mbuffer and if it is on a secure network you can avoid the encryption step.


2

If there is any chance to get high success ratio when deduplication, I would use something like borgbackup or Attic. If not, check the netcat+tar+pbzip2 solution, adapt the compression options according to your hardware - check what is the bottleneck (CPU? network? IO?). The pbzip2 would nicely span across all CPUs, giving better performance.


2

You are using RedHat Linux, so this wouldn't apply, but as another option: I've had great success using ZFS to hold millions of files as inodes aren't an issue. If that was an option for you, you could then take snapshots and use zfs to send incremental updates. I've had a lot of success using this method to transfer as well as archive data. ZFS is ...


2

(Many different answers can work. Here is another one.) Generate the file list with find -type f (this should finish in a couple of hours), split it to small chunks, and transfer each chunk using rsync --files-from=....


2

Try inserting a space before the destination, like this: robocopy "c:\transfer_this.txt" "z: \this.txt" notice the space after the destination "folder" z:.


2

Something like: ssh user@serverB nohup wget -bqc ftp://path/largefile.tar.gz wget options: -b : run in background -q : quiet -c : resume broken download (means you can restart if it breaks) This runs wget in the background so (hopefully) if you exit the ssh shell it'll keep going. Ok, I think you need nohup to ensure that is the case when/if you ...


2

For a first look, you can open the task manager and see if the DISK column shows high load. For a more detailed investigation, you had to use the Performance Monitor application. Follow these steps: open it under "performance monitor", click add (+ sign) and monitor these two counters: Cache/dirty pages: the amount of the write cache, in 4KB pages ...


1

Using tmux/Screen would be a more preferable way instead of nohup. You can always reattach the terminal, in case you loose the connection. For the file transfer itself I would recommend using SSH+Rsync. Rsync can resume files transfers, and it will be encrypted too. Try something like: rsync -av --partial server1:/my/dir server2:/this/dir


1

If it's not very sensitive data and your connection is safe enough, ssh into B and download straight from A via ftp. SSH will make your download considerably slower because of the encryption work overhead. If possible split the 100GB file in multiple ones, especially if the ftp server on A doesn't support download resume.


1

If the destination VMs aren't yet built, you might try using the free VMware Converter to copy the data over. In fact, even if they are built, you could clone the disks to a dummy VM then attach them to existing VM once the clone is done. In any event, the converter uses two methods to clone files from source to destination, the full details of which can ...


1

In Windows 7 there is a ssh.exe Here is what worked for me: 1. create identity (on windows) c:\>ssh-keygen That created an identity file in the home directory. I changed the name of the public key to "id_rsa" 2. copy the file to the target linux system using the ssh Credits to http://serverfault.com/users/984/zoredache for his answer c:\>ssh ...


1

Have you not thought of exposing the SAN LUNs directly to the new VMs - this generally works just fine and can be faster than copying the files into a VMDK - though it can 'lock' the VMs onto their initial host. But you could use this to get things going then migrate the files into a VMDK at your own pace - with rsync - and later cut the link to the original ...


1

Start an rsync daemon on the target machine. This will speedup the transfer process a lot.



Only top voted, non community-wiki answers of a minimum length are eligible