Right, we moved one of our main sites onto a new AWS box with plenty of grunt as it would allow us more control that we had before and future proof ourselves.

About a month ago we started running into issues with high memory usage at the same time every day. In the morning an export is run to export data to a file which is the FTPed to a local machine for processing. The issues were co-inciding with the rough time of the export but when we didn't run the export one day, the server still ran into the same issues. The export has been run at other times in the day since to monitor memory usage to see if it spikes. The conclusion is that the export is fine and barely touches the sides memory wise. No noticeable change in memory usage.

When the issue happens, its effect is to kill mysql and require us to restart the process. We think it might be a mysql memory issue, but might just be that mysql is just the first to feel it. Looking at the logs there is no particular query run before the memory usage hits 90%.

When it strikes at about 9:20am, the memory usage spikes from a near constant 25% to 98% and very quickly kills mysql to save itself. It usually takes about 3-4 minutes to die.

There are no cron jobs running at that time of the day and we haven't noticed a spike in traffic over the period of the issues. Any help would be massively appreciated! thanks.

  • Are you saying that MySQL's memory usage increases using up all of the available memory? – HeatfanJohn Sep 3 '12 at 22:19
  • what size is the ec2 instance? – Mike Sep 3 '12 at 22:36
  • I don't think we have any dedicated mysql monitoring software on the server. Mysql is always the process to be culled though once memory usage hits ~100%. Any recommendations on mysql memory monitoring software? Its a small EC2 – Sam Parmenter Sep 3 '12 at 22:44
  • try mytop. like top, but for mysql. – Sirex Sep 4 '12 at 0:46
  • if you don't even have any graphs, try starting with cacti -- there are some mysql templates pre-built. – Gavin Towey Sep 4 '12 at 2:14

Try using pt-stalk from Percona Toolkit to record detailed system information when this is happening. There's really not enough information here to even begin to troubleshoot your issue.

  • I thought I might be asking a little much from you guys with the information I have given. Thanks for all the advice though, I will install some of the above and return with more information in hand. Thanks guys. – Sam Parmenter Sep 4 '12 at 11:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.