0

I have a question regarding connection reliability running a tcp server from an ec2 instance.

We are currently serving mobile customers around the world from the Oregon region using a c3.4xl ec2 instance. Our product is a live game server written in python using the gevent framework. Right now we serve about 200 - 300 customers concurrently.

The issue is that we have a lot of customers from the other side of the world that are having trouble connecting and staying connected to the server. The server consistently has the clients time out without closing the socket. We're seeing times of > 30s without hearing back from a heartbeat.

Is it wrong of us to assume that a mobile client can establish a long term tcp connection from around the world and have it not be interrupted?

If so, what would be the best way to mitigate this problem?

If not, does anyone have any strategies for debugging the lost connections?

Thanks in advance :)

Improve this question
2
  • Let me tell you, cellular networks outside of technological places such as the US, maybe Hong Kong as well... They suck. Is it viable to increase the timeout period? People on their mobiles are used to waiting.
    – Vasili Syrakis
    May 7, 2014 at 23:22
  • We could increase the timeout but it seems that once we're over 30s it's not coming back. Most likely the connection has gone awol. Once we prompt the user to reconnect they are able to establish a new connection with the server.
    – Zach
    May 8, 2014 at 18:32

2 Answers 2

Reset to default
2

Yes, it's very wrong to assume that TCP is going to be completely reliable. You need to design your application with fault tolerance in mind. TCP will break, timeout, and otherwise behave poorly given the vast array of client devices out there.

How you fix this depends greatly on your application, and is very off topic for serverfault. You'd probably have better luck on stack overflow, or the gamedev stack exchange.

Improve this answer
2
  • Thanks devicenull, that's what we were assuming. We really just wanted to make sure there wasn't something setup incorrectly on our servers and that TCP is inherently going to have issues. As for fixing it within the application layer that's something we can do and we have some ideas for mitigation. Do you know roughly the reliability of TCP or is that impossible to ballpark. Currently about 1/7th of our daily active users have had an issue with a lost connection.
    – Zach
    May 8, 2014 at 18:24
  • Mobile customers are going to be especially bad. I'm not aware of any decent numbers on it, but what you're describing doesn't sound terribly unreasonable.
    – devicenull
    May 9, 2014 at 2:23
1

Spin up a micro or m1.small instance in an AWS region close to your end-users, with HAProxy installed on it.

Configure the proxy in TCP mode to listen on the appropriate port and relay the connections over to Oregon.

The proxy will actually be managing 2 separate connections for each session, one in each direction (from the user, and to your server) and you may find this setup helps stabilize things. The proxy will listen for connections, and each time one comes in, it will make a separate connection outbound to your server. Once that connection comes up, the proxy will blindly tie the data pipes from those connections together and hold the connections up until one end or the other drops -- or the proxy's internal idle timeout timer expires, which will also close the connections, so you may need to increase the timeouts from their default values.

Theoretically, this should not matter, but in practice, the quality and reliability of the connections between the users and the proxy -- and between the proxy and your main server -- may be much better than the "direct" connections, making end-to-end connectivity more reliable.

You should find that HAProxy can handle hundreds of simultaneous connections on a very small server. It's not the only tool for this application but it's the one with which I'm most familar.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .