Tell me more ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.
up vote 1 down vote favorite

I am having issues with tcp connections between two instances in EC2. I thought at first it was the tools I was using (JRuby on Rails stack + MongoDB) when I saw exceptions such as the following in my code:

A Mongo::OperationFailure occurred in foo#bar:

Mongo::OperationFailure
.bundle/jruby/1.8/gems/mongo-1.6.2/lib/mongo/util/tcp_socket.rb:76:in `read'

So thinking this is a software issue, I didn't come to ServerFault. I thought the IO classes in JRuby might be hosed after some research but that wasn't accurate. I went ahead and installed Ruby 1.9.3 and moved the entire stack over to it. Sure enough after a while, a similar exception creeped up:

A Errno::ETIMEDOUT occurred in anotherfoo#anotherbar:

Connection timed out
mongo (1.6.2) lib/mongo/util/tcp_socket.rb:70:in `readpartial'

The reason I come to serverfault for help is because I do believe this might be some inter zone timeout issue in Amazon's infrastructure and I was wondering if any can verify or give me suggestions for how to further debug this as I'm running out of solutions. My app server is in us-east-1a. The mongodb server is in us-east-1c. Perhaps that could be a reason? Why might I bet getting these timeouts using a default Amazon Linux AMI (64-bit, XLARGE)?

share|improve this question

1 Answer

up vote 2 down vote accepted

Newsflash: networking is unreliable. Whether it's EC2 or local colo, sometimes your network won't perform as you might like it to. If your code isn't capable of handling that, you will have problems no matter where you're hosting.

That being said, EC2 availability zones are geographically dispersed, so it's unreasonable to expect that the networking will be as reliable as a LAN (or even within the same AZ). Moving things into the same AZ might improve your reliability, but not to the point that you can hope to get away with code that doesn't take the occasional network hiccup into account. So, fix your code so that it catches appropriate exceptions and retries the failed operation.

share|improve this answer
I believe you are mixing terms. EC2 regions are geographically disperse, but availability zones are not. AZs within a region are units of administrative separation, with separate infrastructure, power, cooling, etc. The idea is that if one AZ in a region has an issue, it shouldn't affect any of the other AZs. Other than that, good answer re: unreliable networking. – EEAA May 6 '12 at 3:10
1  
Each AZ is a separate facility, and sometimes not just "the next warehouse along", but separate by a couple of miles (two of the us-east facilities are apparently on opposite sides of a big ridge, which was important when the hurricane was threatening last year). I got that info straight from AWS engineers, by the way. Whether "a couple of miles distant" counts as "geographically dispersed" or not is, I guess, a matter of opinion. – womble May 6 '12 at 3:25
Ahh, got it. My understanding was wrong then. Thanks for clarifying. – EEAA May 6 '12 at 3:35
Hey @ErikA, if the answer provided clears up the question, consider accepting it? – Mike Fiedler May 6 '12 at 13:33
1  
@MikeFiedler - while I appreciate the initiative you're taking to keep the site rep system running, you're barking up the wrong tree. I'm not the OP and as such, I can't accept womble's answer. You need to talk with imaginative. – EEAA May 6 '12 at 13:53
show 2 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.