Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I am migrating my server from the USA to the UK from one data center to another, my host said I should be able to achieve 11 megabytes per second.

Operating system is Windows Server 2008 at both ends.

My average file size is around 100MB and the data is split across 5 2TB drives.

What would be the recommended way to transfer these files?

  • FTP
  • SMB
  • Rsync / Robocopy
  • Other?

I'm not too bothered about security as these are public files anyway, just want a solution that can push the full 11 MB/s transfer rate to minimize the total transfer time.

share|improve this question
18  
11 MB/s or 11 Mb/s? –  wim Oct 4 '11 at 2:11
12  
transfer the data to binary punch card and use a carrier pigeon :) –  enterzero Oct 4 '11 at 2:13
8  
You should provide detail. How many carrier pigeons do you think it would take? Show your work. –  Evik James Oct 4 '11 at 2:31
16  
@Evik European or African? –  wim Oct 4 '11 at 4:03
8  
As an aside, Wolfram Alpha is the most convenient way to do the calculation, "10 TB at 11MB/s". wolframalpha.com/input/?i=10+TB+at+11MB%2Fs –  pufferfish Oct 5 '11 at 18:33

12 Answers 12

up vote 164 down vote accepted

Ship hard drives across the ocean instead.

At 11 Mbps with full utilization, you're looking at just shy of 90 days to transfer 10 TB.


11 Mbps = 1.375 MBps = 116.015 GB/day.

10240 GB / 116.015 GB/day = ~88.3 days.

share|improve this answer
39  
+1 for Sneakernet. Also, you forgot TCP/IP overhead. It's more like ~100 days under ideal circumstances. –  Chris S Oct 3 '11 at 20:25
41  
A wise man once said "Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway". This equation is very true and not substantially altered by changing the station wagon for a boat. (bpfh.net/sysadmin/never-underestimate-bandwidth.html) –  RobM Oct 3 '11 at 20:36
5  
It's better to ship tapes, or blueray disks, rather than drives. If you go with drives, make sure the originals are kept safe and available just in case. I'd go for the drives myself (unless I had Ultrium 4 drives) because 10 TB = 410 single layer blueray disks! –  Allen Oct 3 '11 at 20:44
9  
Just realised that i typed 11Mbps, however is what I actually meant was 11MB/s. I suppose this makes quite a big difference, my calculations have it to around 11-14 days roughly...is this correct? –  Paul Hinett Oct 3 '11 at 21:14
17  
still believe that sending a man oversee with the 10TB backup while the official disk are still working then once the setup is done, you can lunch a rsync to update the new server for any change. You'd have your machine up and running in about a day. –  Loïc Faure-Lacroix Oct 3 '11 at 22:54

Rsync of course.

At least you can continue at any time after break , and its without any pain.

share|improve this answer
7  
3+ months to copy at 100% utilization. Sorry, but that's a terrible way to transfer that much data. –  Chris S Oct 3 '11 at 20:26
    
I have to agree with @ChrisS, using rsync just to copy large files is not efficient. For my stuff I ended up using tar over netcat or ssh for the initial transfer. It is much faster and starts to transfer immediately, while rsync will scan all files first which takes time. If this get interrupted you still can use rsync afterwards. In fact, I do this sometimes after tar anyway to ensure all permissions, socket files, etc. are correct. –  Martin Scharrer Oct 4 '11 at 7:20
1  
After the OP correcting that he's got ~100Mb connection, not 11Mb, rsync makes much more sense. +1 for the first to mention it. –  Chris S Oct 4 '11 at 12:29

I'd say rsync, at 11 MB/s you will look at 10-14 days and even if you get interrupted, rsync will easily start where it stopped last time.

At 11 Mbps I'd ship the hard disks like suggested above :)

share|improve this answer
1  
Your estimate differs very significantly from what others have posted (and I don't know who is correct). Can you supply your methodology for arriving at those figures? –  John Gardeniers Oct 3 '11 at 23:54
9  
The difference arises from the OP misstating 11 Mbps when in fact he meant 11 MBps -- which is 8 times faster. BTW, restarting a 10 TB rsync in the case of an interruption will probably take a while, won't it? Hours, or longer? –  Frank Farmer Oct 4 '11 at 0:15
    
@FrankFarmer: i wouldn't worry about rsync restarting; I keep an offsite copy of ~20TB over a 30Mbps wireless line, and restarting is in the seconds range. the initial copy took a couple weeks, but the nightly update is usually a couple hours. –  Javier Oct 4 '11 at 4:23
    
@FrankFarmer - rsync seems to scale very well. I have an ~2TB over a rural ADSL1 line that was initalised with sneakernet, but takes ~5 min to rsync every night if nothing has changed. –  Flexo Oct 4 '11 at 7:28
6  
rsync restart time scales with number of files (mainly from stat time, in my experience), not with total data. I would expect no significant wait (several minutes at most). Though my experience with rsync tops at a little under 5TB. –  derobert Oct 4 '11 at 14:52

11Mbps? This is quite a limitation you have here. In your situation I would simply:

  • Clone the data
  • Compress it
  • Rent servers on both ends with at least 10 times more bandwidth (in the same data centers or on your end in a data center near you).
  • Transfer the files
  • Apply the data to the new server.

If you really have no solution to increase bandwidth... Then shipping a physical drive will be way faster.

From my painful experience hard drives tend to break in the mail... USB flash drives are a way better solution for frequent data transfers. In your case it would require a few of them :) So send 2 copies of your data on multiple hard drives.

Considering the amount of data you have you could also send drives from a RAID 5 or RAID 6 array if you have the same hardware/software on the other side to plug your drives in. But in that case remember to mark the order of your drives and their serial numbers so when reconfiguring they don't get mixed-up.

share|improve this answer
    
sorry, the 11Mbps was a mistype, it is 11MB/s...i did mention in one of the above comments. –  Paul Hinett Oct 4 '11 at 19:17

Have you considered IPoAC?

A single pigeon may be able to carry tens of gigabytes of data in around an hour, which on an average bandwidth basis compares very favorably to current ADSL standards, even when accounting for lost drives.

share|improve this answer
18  
Pigeons would suffer signal loss at the distance described by the OP. –  Roy Tinker Oct 4 '11 at 17:03
    
@RoyTinker Cleared IPoAC needs be implemented using a windowing process. –  JamesBarnett Dec 29 '11 at 6:25

I know this is already accepted but have you considered taking your disks to a data center/provider/host where you can get more bandwidth? It'll probably cost you some money but copying 10240Gb to backup disks and sending of will also cost both time and money (2 x money).

Also you'll be sure your disks don't break in transport.

share|improve this answer
    
How is this answer any different from the accepted answer? –  Chris S Oct 4 '11 at 12:26
2  
@Chris This answer suggest transporting the disks to a larger pipe on the same continent. –  Alexandre Jasmin Oct 4 '11 at 18:57

While I have to agree on the "ship it using harddrives" answer in this case, here a copy solution I use when I have to copy large amounts of files for the first time:

While rsync is good to keep two data storages in sync, it introduces quite a bit of unnecessary overhead for the initial transfer. I figured that the fastest way is to tar which gets piped over netcat. On the receiver site you can also use netcat in listen mode which pipes the incoming data to an extracting tar. The benefit is that tar starts sending immediately and netcat sends it as plain TCP stream with no extra higher-level protocol overhead. This should be as fast as it gets. However, it is not simple possible to restart a interrupted transfer at the last position.

It is also easily possible to compress the data for the transfer by using the right tar options or add a compression tool in the pipes. Note that netcat sends the date unencrypted. In cases where this is not an option, an encrypted ssh connection can be used instead (tar <options> | ssh <target> -c 'tar -x <options>').

If all data is transfered rsync can be used to ensure that all files which got updated in the meantime are synchronized. Also IIRC tar doesn't create sockets which will get lost otherwise, but they aren't really used for datacenter data anyway.

share|improve this answer
    
The downside is that it's not tolerant of interuptions –  Joel Coel Oct 18 '11 at 17:50

You should use rsync. It will compress the data and de-duplicate it before sending. It can also resume partial transfers, which is very important for any large transfers.

Its likely it doesn't transfer 10TB; if its logs and text and such it could well be under 1TB; perhaps way below 1TB.

There are tools that do a better job of compression than rsync and likely find more matches. You could use lrzip etc.

There are specific types of data that doesn't compress well and doesn't contain literal dupes - videos and other media for example. In those cases, FTP and rsync are doing much the same effort.

share|improve this answer
3  
RSync deduplicates data? I think it only does this at the file level, meaning deduplication is mostly useless in this case. –  devicenull Oct 5 '11 at 2:41

Never underestimate the bandwidth of a station wagon full of tapes

-- Trad.

In your case, disks or tapes sent by courier, but the principle still applies. If you're not concerned about latency, this will be vastly cheaper than the network bandwidth to transfer 10TB of data in any reasonable length of time.

share|improve this answer
    
Jeff Atwood ran the numbers in one of his old Coding Horror posts.. codinghorror.com/blog/2007/02/the-economics-of-bandwidth.html –  tardate Oct 4 '11 at 19:34

Again, first suggestion is to ship the drives.

Second suggestion is to use rsync to rsyncd, not over SSH. I've tried many things and it is usually the fastest. Remember to turn on compression. Also, look at increasing or decreasing the rsync buffer size to get the optimal transfer rate. It may also help to increase your MTU size. This only helps if routers en route don't fragment your packets though. There are ways to determine if they do.

Unfortunately there is no setting that's always the best. You'll have to experiment to find out what works best in your situation.

share|improve this answer

You can use a torrent for this.

Create a private torrent at one end and use the client on the other.

Although there is encryption in place you must check with your requirements.

share|improve this answer
1  
A 1 to 1 torrent relationship is no better than a 1 to 1 file transfer. If there is limited pipe between the two sites you need multiple seeders on different pipes, ideally geographically distributed. –  Jeremy Oct 5 '11 at 17:43
    
@Jeremy - it's no better or worse in terms of throughput. It may be better in terms of reliability (easy pause/resume), which for this size xfer could be important –  Joel Coel Oct 18 '11 at 17:52

You mentioned the servers are running Windows 2008. Would Microsoft DFS be suitable? There is some magic in the lower end that tries to get as much bandwidth out of the connection as posible, and also has compression and de-duplication (IIRC).

Mind you, hard drives, DVDs or BluRays would be faster... My calculation is 11 days at the full 11 MB/s...

share|improve this answer

protected by sysadmin1138 Oct 5 '11 at 12:04

Thank you for your interest in this question. Because it has attracted low-quality answers, posting an answer now requires 10 reputation on this site.

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.