2

Please, excuse my horrible english. I'm triyng to find the way to clone a HDD server to a remote destination location, using SSH and, if possible, compress data on-the-fly to minimize unnecessary data transmission between them. The main objective is a server migration between physical & VPS appliances.

Broadband is approx 50 - 60 MB/s (transfers at 6-8 MB/s). Running a full-dd-copy estimate a 28 hours copy time. I want reduce that time as much as possible without reduce partition size in source server (i want source as-is).

Origin & destination servers are in offline status (both initiated in rescue mode).

Data on original server are 60 GB. Rest of partition is filled with zeros.

There is a way to archive that?

Until now i cloning entire disks (but not so large) with this command:

dd if=/dev/sda status=progress bs=10M conv=fsync | ssh user@ip dd of=/dev/sda

That works, but this time i'm trying to do it with too much data...

Any ideas?...

Thank you very much in advance to all. Best regards!

D.

Improve this question
9
  • aslong as you don't describe the network and other related things, it will only opinion based things possible
    – djdomi
    Jul 5, 2022 at 16:55
  • Hi, djdomi... What things do you need to complete the question?... Source & destination servers are In different data centers from same enterprise. Have root access in both servers...
    – Diego Ferrarese
    Jul 5, 2022 at 17:44
  • hm, why not use nc for that job, it will pipe the raw data without the hazzle of encryption, second option is use a temporary space to dump the file
    – djdomi
    Jul 5, 2022 at 20:25
  • I was thinking about it, but SSH is better because data encryption. Precise, i don have so much space to temporary dump data, and time factor is something important in this scenario...because of that i thinking sending data with -C flag in ssh, compressing data on-the-fly... but also i'm unsure about conv=fsync in DD... or perhaps conv=sync?... my doubts are in copy of "assigned blank space" more efficiently in network...
    – Diego Ferrarese
    Jul 5, 2022 at 21:06
  • What is the file system? I mean, using tools like e2fsimage might reduce the transfer to only used data blocks. But those tools are, of course, file system dependent.
    – Nikita Kipriyanov
    Jul 6, 2022 at 3:19

2 Answers 2

Reset to default
4

Ok, guys... finally, server migration was executed with status SUCCESS! Data copy time was aprox 5hs 30Min. Command & total time:

dd if=/dev/sda bs=5M conv=fsync status=progress | gzip -c -9 | ssh user@DestinationIP 'gzip -d | dd of=/dev/sda bs=5M'
751593062400 bytes (752 GB, 700GiB) copied, 19185.8 s, 39.2 MB/s

I want to thanks to all for your comments, specially Tom Yan. Hope this help someone else! Best regards!

Improve this answer
0

Ok Guys. I return with new findings. Finally, I think i got it. Thanks to Tom Yan's directions, i realized that piping gzip -c at source side, and gzip -d at destination side did the trick.

I tried -C flag in ssh connection, with relatively no impact in speed.

So, after several tests this was the command that resolved my problem:

dd if=/dev/sda bs=4MiB conv=sync status=progress | gzip -c -9 | ssh user@DestinationIP 'gzip -d | dd of=/dev/sda bs=4MiB'

As Tom Yan's post, firstly (my bad) i not quote whole command line for the remote host and the the result not changed... so, i don't understand the reason of that quotes... of course, finally i put them... but not understand why...

Improve this answer
3
  • If you do not quote the command line for the remote side, the local shell will interpret the | in it, which means you would be piping the output of ssh user@DestinationIP gzip -d to dd of=/dev/sda bs=4MiB run on the local host. (Luckily the source and destination are both /dev/sda, so hopefully that didn't cause you any unexpected data loss...)
    – Tom Yan
    Jul 7, 2022 at 21:31
  • Instead of quoting, you can also use \ to escape any | or > (that are after ssh user@DestinationIP). In that case the command line for the remote shell will remain being passed as multiple arguments (instead of one single argument) to ssh. (I'm not that familiar with ssh to tell whether it could make a difference in certain cases.)
    – Tom Yan
    Jul 7, 2022 at 21:37
  • 1
    Oh! Excellent explanation! Thank you very much! Tomorrow night will executed real job (hope successfully). Thanks again & come back with final results! Regards! D.
    – Diego Ferrarese
    Jul 7, 2022 at 21:58

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .