Over the course of 3 minutes, our RDS instance went from 50GB free down to only 900MB free, before suddenly climbing back up to 50GB. This happened around 9 AM when there wasn't much database activity. Our CPU usage did spike to 60% at same time.

RDS Free Storage

The whole thing happened so fast that it was over by the time we got the notification e-mail and logged into the AWS console.

Digging back further, we had a similar incident 8 days ago, however free disk space didn't go all the way to zero - only to 11%, so our alarm wasn't even triggered.

I'm at a loss as to figuring out what could be causing this. I asked coworkers and none of us were doing any manual queries. None of our write-heavy scripts run during the morning (and they don't cause this much of a drop). I've checked the slow query log and while we have a lot in there, nothing is out of the ordinary. The error log only contains 1 error that we've had for awhile. Even the reported uptime is 64 days, so we know MySQL itself didn't restart.

Do any of you have suggestions on where else I can look to diagnose this and figure out what happened?

  • Can you view the MySQL error log? – Wilson Hauck Mar 21 at 18:57
  • Yes, it only has one error that we've had for awhile: 2019-03-21T14:50:02.888051Z 0 [Note] InnoDB: page_cleaner: 1000ms intended loop took 4502ms. The settings might not be optimal. (flushed=200 and evicted=0, during the time.) (We've had this for over a year and I can't seem to fix it so we've been ignoring it; doesn't seem to affect anything) – DOOManiac Mar 21 at 21:46
  • SHOW VARIABLES LIKE 'innodb_%'; – Rick James Apr 16 at 21:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.