I have a t3a.micro instance running wordpress with pretty low traffic. This instance automatically becomes unavailable resulting in 504 Gateway timeout error. At that moment I fail to connect with EC2 using ssh as well. It happens anytime in day, sometimes daily and sometimes doesn't happen at all for whole week. There is no traffic spike when it goes down as well. I have asked this question from AWS Support as well and got following answer:
Upon checking from my end, I was able to see that the instance has been failing the instance status check several times in the past 24 hours indicating that the instance has issues at the OS level. Upon further checking the console logs, I was able to see the following Out of Memory error:
[ 5626.002561] Out of memory: Killed process 2050 (apache2) total-vm:552924kB, anon-rss:54868kB, file-rss:0kB, shmem-rss:32076kB, UID:33 pgtables:848kB oom_score_adj:0
[ 5674.174673] Out of memory: Killed process 1788 (apache2) total-vm:624936kB, anon-rss:51952kB, file-rss:0kB, shmem-rss:34184kB, UID:33 pgtables:856kB oom_score_adj:0
[ 5763.820732] Out of memory: Killed process 1815 (apache2) total-vm:550384kB, anon-rss:51604kB, file-rss:0kB, shmem-rss:34532kB, UID:33 pgtables:836kB oom_score_adj:0
[ 5773.275938] Out of memory: Killed process 1973 (apache2) total-vm:624744kB, anon-rss:52260kB, file-rss:0kB, shmem-rss:32136kB, UID:33 pgtables:856kB oom_score_adj:0
[ 5959.347157] Out of memory: Killed process 2014 (apache2) total-vm:552440kB, anon-rss:54020kB, file-rss:0kB, shmem-rss:28856kB, UID:33 pgtables:844kB oom_score_adj:0
[ 6438.787255] Out of memory: Killed process 2165 (apache2) total-vm:624756kB, anon-rss:51948kB, file-rss:0kB, shmem-rss:29836kB, UID:33 pgtables:856kB oom_score_adj:0
A bit about OOM (Out of memory) killer, if memory is exhaustively used up by processes, to the extent which can threaten the stability of the system, then the OOM killer kicks in. It is the task of the OOM Killer to continue killing processes until enough memory is freed for the smooth functioning of the rest of the process that the Kernel is attempting to run. It seems from the output that your instance might have exhausted the memory, thus the OOM killer is being called upon by the Linux Kernel.
They suggested to monitor memory usage by each process and kill or fix that process to not use that much memory. This doesn't look like a very practical approach at a time when I am running wordpress, that I can not modify even if it using too much memory for few minutes.
I have another instance t3.small using elastic beanstalk that hosts a java web application with nginx/tomcat environment on amazon linux AMI provided by beanstalk. Same issue happens with this instance, automatically goes down and shows 504 Gateway timeout.
Question: - I don't want to upgrade my instance because they are running perfectly fine with current amount of traffic on them. Is there any solution to handle these issues without upgrade and ofcourse constant monitoring of processes?