0

I have this t2.micro EC2 instance on AWS running two experimental REST services and MySQL inside Docker, receiving REST calls from sources external to EC2. One instance is HTTP on port 81 the other is HTTPS on 443.

Occasionally the CPU shot up to 100% based on CloudWatch but without much spike in network or disk activities. The result of these were that I was unable to SSH into this instance each time.

My local logs to text file does not show much that the REST server was doing much. What could be the causes and how do I even begin to diagnose the problem? If I can't SSH into this machine is it possible to know which process is using up that CPU cycles?

EDIT: BTW, the EC2 console did say the instance was still green when all this happened — running.

1
  • huggie t2.micro has 1G RAM and 1 CPU did you configure minimums for most of your 2 MySQL instance my.cnf server system global variables? The following URL lists everything but production for the planned use of t2.xxxxx instances. aws.amazon.com/ec2/instance-types/t2 – Wilson Hauck Jun 17 '20 at 22:21
1

A few ideas, not sure either will work perfectly, but if no-one else has a better idea they might help a bit.

  1. Try Monit. Configure it to monitor the key processes as per the documentation, keep the page open, the page refreshes every 60 seconds. Example below.

  2. Stay logged in via SSH with top running.

  3. Use System Manager Session Manager for an alternate way of SSHing in. It uses https from the instance to AWS infrastructure to access the machine and give you an ssh console which might work when standard ssh doesn't. It's somewhat tricky to set up, you have to get your security groups and VPC endpoints set up correctly which can take a bit of time. I've automated it with CloudFormation but I can't easily extract that out of my templates, someone will have created a template you can use - look on git hub.

Monit Example

1

There are multiple ways you could tackle this:

  1. Enable detailed CloudWatch monitoring to get more insights in order to understand what is happening (my assumption is that you are running out of memory and then the OOM killer kicks in)
  2. Check the Systems Logs for that instance by navigating to "Actions" -> "Instance Settings" -> "Get System Log" to find out what is actually happening
  3. Use a bigger instance type so that it will not run out of resources and will allow you some time to analyze what is the consumer, until it gets again unstable.
3
  • The OOM killer you mentioned? Does it kill the instance? My was still green when all this happened. – huggie Jun 17 '20 at 8:52
  • 1
    The OOM killer randomly kills processes to free up memory. You can read more about it [here|en.wikipedia.org/wiki/Out_of_memory]. So in this case, it might kill the ssh daemon and you would not be able to ssh into the instance. – pincoded Jun 17 '20 at 9:28
  • I see. When CPU spikes up to 100% my REST service is inaccessible as well. Clients look like experiencing long time outs. They just hangs there. If service was killed port would not be open I'd thought it would immediately return – huggie Jun 17 '20 at 10:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.