0

My EC2 instance (t2.small) stopped accepting connections on SSH or its other service, but the EC2 control panel said the automatic status checks had not failed, even after several hours. I could not reboot it using the control panel, but I could stop it and start it again. It became unresponsive twice more that day.

After that, I configured cgroups to limit the CPU and memory usage of a mildly resource-hogging process, but that doesn't seem like the right answer. That shouldn't have brought the machine to a halt. (It has no swap, but the OOM killer should have simply killed a process if the instance was out of memory.) "Get System Log" and "Get Instance Screenshot" showed nothing suspicious. The server is running some fairly trusted software, like postfix and gitolite, plus an in-development server which was running as a user. When I look at the CPU usage graph, it shows 2.5% during that time, with occasional spikes to around 6%.

What can I do to figure this out and prevent it from happening again? All I can think is that it was a hardware issue, but I thought that would have been very unlikely.

  • what instance size is it? – Mike Nov 21 '17 at 11:25
  • @Mike It's t2.small. I'l add that to the description. – piojo Nov 21 '17 at 11:48
  • the t2 might be the issue.. you are almost last in pecking order for CPU requests.. i you stop/start the instance it might move to a quieter host – Mike Nov 21 '17 at 14:11
  • @Mike The problem didn't seem to be transient. The first occurrence was on the weekend, and it stayed stuck until I stopped/started it on Monday. (It's not yet critical that it stay up, but it will be soon.) – piojo Nov 22 '17 at 3:18
  • @piojo were you ever able to find an answer to this? – voxobscuro Sep 5 '18 at 22:14
2

I ran into a similar issue and eventually found a solution in looking at the CPU credit usage of my t series instance. By default t series instances run on CPU credits, and once they are exhausted the instance becomes unresponsive until more credits are awarded.

For me, unresponsive meant: unable to access any services and websites provided by the instance, including unresponsive to ssh connection attempts. Attempts to reboot the instance failed to revive it, and only stopping the instance and starting it again would recover it.

I don’t know the semantics of the CPU credit system regarding power off/on events, but I assume that a power cycle must itself come with a credit award.

The solution for preventing a t2 or t3 instance from becoming unavailable in this way is one of:

  • enable t2 unlimited (or t3 unlimited), possibly for an extra cost
  • upgrade to a fixed allocation instance (m series), in my case this would have doubled the cost of the VM as the upgrade path was from a t2.medium ($0.045/hr) to an m5d.large ($0.113/hr).
  • re-deploy to a new t series instance, which comes with a new allocation of credits. Cost is subtle here, booting a fresh VM comes with some cost but I don’t know what it is.

t2/t3 unlimited means that once a server has exceeded usage of its CPU credits, additional charges apply for further CPU usage until more credits are awarded. Currently excess usage is calculated over a 24 hour period.

In our specific case, our t2 Jenkins server was becoming unresponsive sometimes on somedays. It’s likely that source code commits on those days were high frequency, causing CI to run more frequently, and burn too many credits. We switched to the new t3, and enabled T3 Unlimited. It’s only been a few days, but the problem has not resurfaced yet. Being a CI server it is unlikely to run much outside of the general working hours and I have strong doubts that the cost will ever actually be greater.

Hopefully this post mortem will help anyone hitting the same snag.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.