0

I want to spawn a redis slave in AWS (Sydney), which needs ~20Mbit/s to catch up on the writes of the master located in germany.

Problem: I get only 8Mbit/s (1MB/s).

Tried bigger instance types after reading Bandwidth limits for Amazon EC2 but still no difference.

Tried to download random content hosted in australia: tadaa, 20Mbit/s on one download.

Then the idea: Start multiple wgets in parallel to the same host in germany. Tadaa: Every download gets 1MB/s, and the speed sums up to more than 150Mbit/s (didn't try to get more, but should have been possible).

How to get rid of the per connection speed limit?

6
  • 1
    Well it doesn't sound like bandwidth. Could be IO-bound reading from origin or writing to destination. Guess redis could have some rate limiting somewhere, too. On that note, you should use iperf instead of wget for bandwidth testing.
    – Peter
    Dec 5, 2014 at 13:48
  • No, its not IO-bound, because parallel downloads all having 1MB/s at the same time were possible (started up to 15 parallel downloads of an ubuntu image hosted in germany, just to test the speed). The speed of all downloads combined was taken with iftop.
    – edlerd
    Dec 5, 2014 at 13:53
  • 1
    That doesn't rule out the IO of reading from the source. Perform an iperf test. This will also rule out bandwidth bottlenecking on the origin outbound.
    – Peter
    Dec 5, 2014 at 13:56
  • Did you try AWS Direct Connect?
    – risyasin
    Dec 5, 2014 at 14:41
  • AWS Direct Connect is not available on our german hoster hetzner. Or is there another way to use it?
    – edlerd
    Dec 5, 2014 at 15:36

1 Answer 1

0

As mentioned in the comments: IO-bound due to the high latency.

For Redis slaves the solution is an ssh tunnel (with compression enabled) as described in this blog post [1]. Due to the compression, bandwidth is cut by a factor of 10, so in our case 1MB/s was enough.

For downloads the solution is aria2c, a cli download client, which can open up to 16 connections in parallel for big files. This way you can join the bandwidth of multiple connections.

[1] http://tech.3scale.net/2012/07/25/fun-with-redis-replication/

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .