Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We are a small company and we will need to transfer rather large amounts of data (10GB+ each time) between Europe and China in the near future.

As many may have experienced, Internet connections to or from China can be rather unreliable and slow at times without any apparent reason. For example, while sending data to China via FTP generally works well, it can be painfully slow in the other direction.

Currently, we are investigating new ways to have high transfer rates in both directions. So far we have tried:

  1. FTP (see above)
  2. FTP over VPN services (generally slower than direct connections)
  3. F2F (like Retroshare or Freenet - slow!!)
  4. Aspera (fast but expensive!)
  5. BitTorrent (unreachable end nodes, b/c of firewalls which we must not configure)

We would like to try:

  1. Cloud storage (e.g. Amazon S3, Google Storage) - are those services always and reliably reachable from inside China?
  2. Point-to-Point VPN (currently not possible, b/c of the network, see above)

I'd be especially grateful to hear from people who have already dealt with this kind of problem before.

share|improve this question
    
WAN Optimization equipment can help in a number of protocols, en.wikipedia.org/wiki/WAN_optimization. –  3molo Apr 13 '12 at 10:54
8  
Never underestimate the bandwidth of UPS –  uSlackr Apr 13 '12 at 11:05
1  
Even though I answered below, I think you should consider @uSlackr's suggestion. At least look at the cost. 16GB USB sticks can be picked up for a tenner and can be re-used. I'm not sure what postage to China is like but the cost depends on your use case; how often you're actually sending this data. Is it worth maintaining a file sharing system for the sake of a couple of transfers a year? –  webtoe Apr 13 '12 at 12:01
    
One word: Sneakernet. –  pfo Apr 13 '12 at 12:45
2  
When you finally settle on something and have it in production, can you come back in 6 months and tell us what you did and how its working? Add it as an answer to this question so we can all learn. –  uSlackr Apr 13 '12 at 13:18
show 4 more comments

3 Answers

I would suggest using rsync to do these sort of transfers. It can use ssh to encrypt the traffic (so you don't need to use a separate VPN), compresses the stream (which can be quite effective depending on your type of data) and can restart transfers if they fail (it uses a rolling checksum to see whereabouts a transfer failed and where to re-start from).

I use the following which incorporates the above (as the rsync client on my/most machines use ssh by default):

rsync -avz <local file> username@hostname:/path/where/to/place/file/<remote file>

You can use --progress to see on the terminal how well the transfer is progressing. Usually it ssh's in and uses the local machine rsync client but there is also a rsync daemon. I have never used it but worth investigating if you want an actual daemon similar to ftpd.

As it uses SSH you can also alter the ports it uses if firewalls lock them down. i.e. you don't have to have ssh listening on port 22 (in fact I don't to avoid random people on the internet scanning my SSH).

It is usually used on Unix systems (which includes Macs) but you can install it with Cygwin as well if you need to use it on Windows.

If you want an actual service, I use rsync.net who, rather obviously, provide storage to which you can rsync data easily to. They have a European data centre in Sweden which would fit your purposes and don't limit in/out traffic speeds (though check the small print).

share|improve this answer
    
Great suggestion! rsync is a great tool! I will look into that, too. –  Nils Apr 13 '12 at 13:00
add comment

I suggest you test a physical shipment of a disk or thumb drive back-forth to China to understand how long & how much it would take to send the data via a shipping company. The bandwidth is HUGE from a data perspective. You can encrypt and send multiple shipments to provide redundancy and protection and you have no infrastructure to maintain.

share|improve this answer
    
I agree, the bandwidth is tremendous especially if you have Petabytes to move. But in our case it is just some GB, which should be doable via Internet shouldn't it? –  Nils Apr 13 '12 at 12:53
1  
You might think, but the complications seem to be piling up and one of your requirement is reliable... –  uSlackr Apr 13 '12 at 12:54
    
So, is that really the way big companies do it? For example, is that how Apple sends their data to Foxconn? –  Nils Apr 13 '12 at 13:06
    
Apple have the budget for Aspera. –  Tom O'Connor Oct 5 '12 at 12:33
add comment

Some of you have requested to know what we settled for, finally.

Though, we have tried and tested almost all of the suggested solutions above, using Aspera's proprietary protocol and software seem to do the best job for our purpose.

Maybe somebody out there else has still knows of something better and would like to share that?

share|improve this answer
    
Aspera does rule. –  Tom O'Connor Oct 5 '12 at 12:34
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.