96

I am migrating my server from the USA to the UK from one data center to another. My host said I should be able to achieve 11 megabytes per second.

The operating system is Windows Server 2008 at both ends.

My average file size is around 100 MB and the data is split across five 2 TB drives.

What would be the recommended way to transfer these files?

  • FTP
  • SMB
  • Rsync / Robocopy
  • Other?

I'm not too bothered about security as these are public files anyway, but I just want a solution that can push the full 11 MB/s transfer rate to minimize the total transfer time.

13
  • 19
    11 MB/s or 11 Mb/s?
    – wim
    Oct 4, 2011 at 2:11
  • 15
    transfer the data to binary punch card and use a carrier pigeon :)
    – enterzero
    Oct 4, 2011 at 2:13
  • 9
    You should provide detail. How many carrier pigeons do you think it would take? Show your work.
    – Evik James
    Oct 4, 2011 at 2:31
  • 18
    @Evik European or African?
    – wim
    Oct 4, 2011 at 4:03
  • 8
    As an aside, Wolfram Alpha is the most convenient way to do the calculation, "10 TB at 11MB/s". wolframalpha.com/input/?i=10+TB+at+11MB%2Fs
    – pufferfish
    Oct 5, 2011 at 18:33

11 Answers 11

173

Ship hard drives across the ocean instead.

At 11 Mbps with full utilization, you're looking at just shy of 90 days to transfer 10 TB.


11 Mbps = 1.375 MBps = 116.015 GB/day.

10240 GB / 116.015 GB/day = ~88.3 days.

29
  • 43
    +1 for Sneakernet. Also, you forgot TCP/IP overhead. It's more like ~100 days under ideal circumstances.
    – Chris S
    Oct 3, 2011 at 20:25
  • 44
    A wise man once said "Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway". This equation is very true and not substantially altered by changing the station wagon for a boat. (bpfh.net/sysadmin/never-underestimate-bandwidth.html)
    – Rob Moir
    Oct 3, 2011 at 20:36
  • 5
    It's better to ship tapes, or blueray disks, rather than drives. If you go with drives, make sure the originals are kept safe and available just in case. I'd go for the drives myself (unless I had Ultrium 4 drives) because 10 TB = 410 single layer blueray disks!
    – Allen
    Oct 3, 2011 at 20:44
  • 10
    Just realised that i typed 11Mbps, however is what I actually meant was 11MB/s. I suppose this makes quite a big difference, my calculations have it to around 11-14 days roughly...is this correct? Oct 3, 2011 at 21:14
  • 18
    still believe that sending a man oversee with the 10TB backup while the official disk are still working then once the setup is done, you can lunch a rsync to update the new server for any change. You'd have your machine up and running in about a day. Oct 3, 2011 at 22:54
26

I'd say rsync, at 11 MB/s you will look at 10-14 days and even if you get interrupted, rsync will easily start where it stopped last time.

At 11 Mbps I'd ship the hard disks like suggested above :)

8
  • 1
    Your estimate differs very significantly from what others have posted (and I don't know who is correct). Can you supply your methodology for arriving at those figures? Oct 3, 2011 at 23:54
  • 9
    The difference arises from the OP misstating 11 Mbps when in fact he meant 11 MBps -- which is 8 times faster. BTW, restarting a 10 TB rsync in the case of an interruption will probably take a while, won't it? Hours, or longer? Oct 4, 2011 at 0:15
  • @FrankFarmer: i wouldn't worry about rsync restarting; I keep an offsite copy of ~20TB over a 30Mbps wireless line, and restarting is in the seconds range. the initial copy took a couple weeks, but the nightly update is usually a couple hours.
    – Javier
    Oct 4, 2011 at 4:23
  • @FrankFarmer - rsync seems to scale very well. I have an ~2TB over a rural ADSL1 line that was initalised with sneakernet, but takes ~5 min to rsync every night if nothing has changed.
    – Flexo
    Oct 4, 2011 at 7:28
  • 6
    rsync restart time scales with number of files (mainly from stat time, in my experience), not with total data. I would expect no significant wait (several minutes at most). Though my experience with rsync tops at a little under 5TB.
    – derobert
    Oct 4, 2011 at 14:52
15

Rsync of course.

At least you can continue at any time after a break, and it's without any pain.

3
  • 7
    3+ months to copy at 100% utilization. Sorry, but that's a terrible way to transfer that much data.
    – Chris S
    Oct 3, 2011 at 20:26
  • I have to agree with @ChrisS, using rsync just to copy large files is not efficient. For my stuff I ended up using tar over netcat or ssh for the initial transfer. It is much faster and starts to transfer immediately, while rsync will scan all files first which takes time. If this get interrupted you still can use rsync afterwards. In fact, I do this sometimes after tar anyway to ensure all permissions, socket files, etc. are correct. Oct 4, 2011 at 7:20
  • 1
    After the OP correcting that he's got ~100Mb connection, not 11Mb, rsync makes much more sense. +1 for the first to mention it.
    – Chris S
    Oct 4, 2011 at 12:29
12

Never underestimate the bandwidth of a station wagon full of tapes

-- Trad.

In your case, disks or tapes sent by courier, but the principle still applies. If you're not concerned about latency, this will be vastly cheaper than the network bandwidth to transfer 10TB of data in any reasonable length of time.

1
10

You should use rsync. It will compress the data and de-duplicate it before sending. It can also resume partial transfers, which is very important for any large transfers.

It's likely it doesn't transfer 10 TB; if it's logs and text and such it could well be under 1 TB; perhaps way below 1 TB.

There are tools that do a better job of compression than rsync and likely find more matches. You could use lrzip, etc.

There are specific types of data that doesn't compress well and doesn't contain literal dupes - videos and other media for example. In those cases, FTP and rsync are doing much the same effort.

1
  • 4
    RSync deduplicates data? I think it only does this at the file level, meaning deduplication is mostly useless in this case.
    – devicenull
    Oct 5, 2011 at 2:41
6

I know this is already accepted but have you considered taking your disks to a data center/provider/host where you can get more bandwidth? It'll probably cost you some money but copying 10240Gb to backup disks and sending of will also cost both time and money (2 x money).

Also you'll be sure your disks don't break in transport.

2
  • How is this answer any different from the accepted answer?
    – Chris S
    Oct 4, 2011 at 12:26
  • 2
    @Chris This answer suggest transporting the disks to a larger pipe on the same continent. Oct 4, 2011 at 18:57
5

11Mbps? This is quite a limitation you have here. In your situation I would simply:

  • Clone the data
  • Compress it
  • Rent servers on both ends with at least 10 times more bandwidth (in the same data centers or on your end in a data center near you).
  • Transfer the files
  • Apply the data to the new server.

If you really have no solution to increase bandwidth... Then shipping a physical drive will be way faster.

From my painful experience hard drives tend to break in the mail... USB flash drives are a way better solution for frequent data transfers. In your case it would require a few of them :) So send 2 copies of your data on multiple hard drives.

Considering the amount of data you have you could also send drives from a RAID 5 or RAID 6 array if you have the same hardware/software on the other side to plug your drives in. But in that case remember to mark the order of your drives and their serial numbers so when reconfiguring they don't get mixed-up.

1
  • 1
    sorry, the 11Mbps was a mistype, it is 11MB/s...i did mention in one of the above comments. Oct 4, 2011 at 19:17
4

While I have to agree on the "ship it using harddrives" answer in this case, here a copy solution I use when I have to copy large amounts of files for the first time:

While rsync is good to keep two data storages in sync, it introduces quite a bit of unnecessary overhead for the initial transfer. I figured that the fastest way is to tar which gets piped over netcat. On the receiver site you can also use netcat in listen mode which pipes the incoming data to an extracting tar. The benefit is that tar starts sending immediately and netcat sends it as plain TCP stream with no extra higher-level protocol overhead. This should be as fast as it gets. However, it is not simple possible to restart a interrupted transfer at the last position.

It is also easily possible to compress the data for the transfer by using the right tar options or add a compression tool in the pipes. Note that netcat sends the date unencrypted. In cases where this is not an option, an encrypted ssh connection can be used instead (tar <options> | ssh <target> -c 'tar -x <options>').

If all data is transfered rsync can be used to ensure that all files which got updated in the meantime are synchronized. Also IIRC tar doesn't create sockets which will get lost otherwise, but they aren't really used for datacenter data anyway.

1
  • The downside is that it's not tolerant of interuptions
    – Joel Coel
    Oct 18, 2011 at 17:50
3

Again, first suggestion is to ship the drives.

Second suggestion is to use rsync to rsyncd, not over SSH. I've tried many things and it is usually the fastest. Remember to turn on compression. Also, look at increasing or decreasing the rsync buffer size to get the optimal transfer rate. It may also help to increase your MTU size. This only helps if routers en route don't fragment your packets though. There are ways to determine if they do.

Unfortunately there is no setting that's always the best. You'll have to experiment to find out what works best in your situation.

2

You mentioned the servers are running Windows 2008. Would Microsoft DFS be suitable? There is some magic in the lower end that tries to get as much bandwidth out of the connection as posible, and also has compression and de-duplication (IIRC).

Mind you, hard drives, DVDs or BluRays would be faster... My calculation is 11 days at the full 11 MB/s...

1

You can use a torrent for this.

Create a private torrent at one end and use the client on the other.

Although there is encryption in place you must check with your requirements.

2
  • 1
    A 1 to 1 torrent relationship is no better than a 1 to 1 file transfer. If there is limited pipe between the two sites you need multiple seeders on different pipes, ideally geographically distributed.
    – Jeremy
    Oct 5, 2011 at 17:43
  • @Jeremy - it's no better or worse in terms of throughput. It may be better in terms of reliability (easy pause/resume), which for this size xfer could be important
    – Joel Coel
    Oct 18, 2011 at 17:52

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .