Tell me Guys, how would you transfer more than one million files, witch have a summarized size of about 150 GB from A servers (KVM) VM to another (let's call it B, XenCenter) servers other VM in maximum 2 days?

Currently I'm trying to compress them to an external Hard drive and then copy them on the B servers VM. However 2 issues have ocured:

  • Only the compression to tar.gz takes about 3 days
  • I'm unable to umount any hard drive from the VM, donno why. (CentOS 6.3) I can mount it up, I can see the files, everything. But after I give the umount command it hangs (waited 1h30m for a 16GB pendrive with one 500MB file to umount....)

The two VMs are unable to see each other and that cannot be changed. I'm currently open for any solutions that comes to your mind.

  • Can you not move the backing store for the virtual disk instead? – Spooler Jun 5 '18 at 14:48
  • Sorry, I can't understand what you mean. – Bert Jun 5 '18 at 14:51
  • If both of these are VMs, they store their data in some form of lower level storage in order to provide virtual disks. Those may be files, logical volumes, or LUNs from a SAN, etc. Depending on how the virtual disks are backed, you may be able to very easily copy those instead. Doing a file based copy of a million files with compression is going to crawl. – Spooler Jun 5 '18 at 14:55
  • 1
    dd or rsync (ext4,xfs) – alexus Jun 5 '18 at 14:56
  • 1
    How about just tar'ing them? Leave off the compression step and just allow tar to bundle all your files up into a single tarball (uncompressed) that can more easily be moved around. Alternatively, unmount then dd the block device to an image on removable storage, then mount on the other side? – dannosaur Jun 5 '18 at 17:23

Oh how simple it is.... a friend of me told me the answer.

Since they are both acting as webservers, I just tar the files without compression, move to the root folder of the website and I can simply download it to the B server with wget..... Yes... sometimes the outsiders have the best option. :)

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.