5

We are a small company and we will need to transfer rather large amounts of data (10GB+ each time) between Europe and China in the near future.

As many may have experienced, Internet connections to or from China can be rather unreliable and slow at times without any apparent reason. For example, while sending data to China via FTP generally works well, it can be painfully slow in the other direction.

Currently, we are investigating new ways to have high transfer rates in both directions. So far we have tried:

  1. FTP (see above)
  2. FTP over VPN services (generally slower than direct connections)
  3. F2F (like Retroshare or Freenet - slow!!)
  4. Aspera (fast but expensive!)
  5. BitTorrent (unreachable end nodes, b/c of firewalls which we must not configure)

We would like to try:

  1. Cloud storage (e.g. Amazon S3, Google Storage) - are those services always and reliably reachable from inside China?
  2. Point-to-Point VPN (currently not possible, b/c of the network, see above)

I'd be especially grateful to hear from people who have already dealt with this kind of problem before.

10
  • WAN Optimization equipment can help in a number of protocols, en.wikipedia.org/wiki/WAN_optimization.
    – 3molo
    Apr 13, 2012 at 10:54
  • 8
    Never underestimate the bandwidth of UPS
    – uSlackr
    Apr 13, 2012 at 11:05
  • 1
    Even though I answered below, I think you should consider @uSlackr's suggestion. At least look at the cost. 16GB USB sticks can be picked up for a tenner and can be re-used. I'm not sure what postage to China is like but the cost depends on your use case; how often you're actually sending this data. Is it worth maintaining a file sharing system for the sake of a couple of transfers a year?
    – webtoe
    Apr 13, 2012 at 12:01
  • One word: Sneakernet.
    – pfo
    Apr 13, 2012 at 12:45
  • 2
    When you finally settle on something and have it in production, can you come back in 6 months and tell us what you did and how its working? Add it as an answer to this question so we can all learn.
    – uSlackr
    Apr 13, 2012 at 13:18

4 Answers 4

3

I would suggest using rsync to do these sort of transfers. It can use ssh to encrypt the traffic (so you don't need to use a separate VPN), compresses the stream (which can be quite effective depending on your type of data) and can restart transfers if they fail (it uses a rolling checksum to see whereabouts a transfer failed and where to re-start from).

I use the following which incorporates the above (as the rsync client on my/most machines use ssh by default):

rsync -avz <local file> username@hostname:/path/where/to/place/file/<remote file>

You can use --progress to see on the terminal how well the transfer is progressing. Usually it ssh's in and uses the local machine rsync client but there is also a rsync daemon. I have never used it but worth investigating if you want an actual daemon similar to ftpd.

As it uses SSH you can also alter the ports it uses if firewalls lock them down. i.e. you don't have to have ssh listening on port 22 (in fact I don't to avoid random people on the internet scanning my SSH).

It is usually used on Unix systems (which includes Macs) but you can install it with Cygwin as well if you need to use it on Windows.

If you want an actual service, I use rsync.net who, rather obviously, provide storage to which you can rsync data easily to. They have a European data centre in Sweden which would fit your purposes and don't limit in/out traffic speeds (though check the small print).

1
  • Great suggestion! rsync is a great tool! I will look into that, too.
    – nils
    Apr 13, 2012 at 13:00
2

I suggest you test a physical shipment of a disk or thumb drive back-forth to China to understand how long & how much it would take to send the data via a shipping company. The bandwidth is HUGE from a data perspective. You can encrypt and send multiple shipments to provide redundancy and protection and you have no infrastructure to maintain.

4
  • I agree, the bandwidth is tremendous especially if you have Petabytes to move. But in our case it is just some GB, which should be doable via Internet shouldn't it?
    – nils
    Apr 13, 2012 at 12:53
  • 1
    You might think, but the complications seem to be piling up and one of your requirement is reliable...
    – uSlackr
    Apr 13, 2012 at 12:54
  • So, is that really the way big companies do it? For example, is that how Apple sends their data to Foxconn?
    – nils
    Apr 13, 2012 at 13:06
  • Apple have the budget for Aspera. Oct 5, 2012 at 12:33
1

Some of you have requested to know what we settled for, finally.

Though, we have tried and tested almost all of the suggested solutions above, using Aspera's proprietary protocol and software seem to do the best job for our purpose.

Maybe somebody out there else has still knows of something better and would like to share that?

1
0

I guess you need distributed file system.

Ceph is a free-software storage platform, implements object storage on a single distributed computer cluster, and provides interfaces for object-, block- and file-level storage. Ceph aims primarily for completely distributed operation without a single point of failure, scalable to the exabyte level, and freely available.

Ceph replicates data and makes it fault-tolerant, using commodity hardware and requiring no specific hardware support. As a result of its design, the system is both self-healing and self-managing, aiming to minimize administration time and other costs.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .