My server crashed 3 times in the last week from out of memory.

  • Server is ubuntu 14.04 with 8Gb ram.
  • MySql server 5.5.50

Below is a graph from munin:

memory consumption

I didn't have the historical ram consumption from munin. Luckily we track it and store it in a database:

enter image description here

Consumption increased steadily over the last few weeks

My main assumption is that mysql is the cause. I am not sure that current usage of our application can explain this increase.

  • How can I know for sure that mysql is the highest consumer of (committed) memory?
  • I am currently checking mysqltuner and the slow log. Is there a way to check which queries consume the most ram?
  • Anything else that you think I should check?
  • Does the load grow similar to your DB's RAM usage? You can check MySQL memory consumption: dev.mysql.com/doc/refman/5.7/en/memory-summary-tables.html – Lenniey Feb 21 '20 at 10:11
  • You can get a quick look at which programs have the largest virtual memory size with ps aux --sort -vsz | head. It seems that there is a large memory commit somewhere. – Gerrit Feb 21 '20 at 11:13
  • @Gerrit you are amazing!! I found some dormant processes that consumed the memory. I killed them, and now memory usage dropped significantly! – justadev Feb 21 '20 at 11:47
  • 1
    Additional information request. # cores, any SSD or NVME devices on MySQL Host server? Post on pastebin.com and share the links. A) complete (not edited) my.cnf or my.ini From your SSH login root, Text results of: B) SHOW GLOBAL STATUS; after minimum 24 hours UPTIME C) SHOW GLOBAL VARIABLES; D) SHOW FULL PROCESSLIST; for server workload tuning analysis to provide suggestions. You have NO swap space apparently. If your server is allowed to hibernate, swap space should be RAMx1.2. If hibernate not allowed, swap should be RAMx.2 for 20%, better to survive than get OOM. – Wilson Hauck Mar 7 '20 at 16:27

In this case there are large commits that are not actively used by apps. You can see that the committed memory is much larger then the active memory.

To quickly find likely culprits, use:

ps aux --sort -vsz | head

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.