Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm running an Ubuntu server for MySQL.

Server info

  • Ubuntu 12.10
  • MySQL installed via apt
  • RAM: 512M
  • innodb_buffer_pool_size : 300M
  • There is no other memory intensive application running on this box.

Problem

Every morning, at approx. 6:40am something happens to cause a noticeable change in memory:

https://dl.dropbox.com/u/12520837/mem.s.png

At the same time, a systematic "kill" of running processes seems to occur, causing MySQL to restart.

Apr 10 06:43:40 mysql-01 kernel: [1866472.511966] select 1 (init), adj 0, size 41, to kill

Apr 10 06:43:40 mysql-01 kernel: [1866472.511973] select 385 (dbus-daemon), adj 0, size 44, to kill

Apr 10 06:43:40 mysql-01 kernel: [1866472.511975] select 389 (rsyslogd), adj 0, size 124, to kill

Apr 10 06:43:40 mysql-01 kernel: [1866472.511982] select 4578 (snmpd), adj 0, size 160, to kill

Apr 10 06:43:40 mysql-01 kernel: [1866472.514157] select 1 (init), adj 0, size 41, to kill

Apr 10 06:43:40 mysql-01 kernel: [1866472.514164] select 385 (dbus-daemon), adj 0, size 44, to kill

Apr 10 06:43:40 mysql-01 kernel: [1866472.514166] select 389 (rsyslogd), adj 0, size 124, to kill

Apr 10 06:43:40 mysql-01 kernel: [1866472.514171] select 4578 (snmpd), adj 0, size 160, to kill

Apr 10 06:43:44 mysql-01 /etc/mysql/debian-start[21807]: Upgrading MySQL tables if necessary.

Apr 10 06:43:45 mysql-01 /etc/mysql/debian-start[21810]: /usr/bin/mysql_upgrade: the '--basedir' option is always ignored

Apr 10 06:43:45 mysql-01 /etc/mysql/debian-start[21810]: Looking for 'mysql' as: /usr/bin/mysql

Apr 10 06:43:45 mysql-01 /etc/mysql/debian-start[21810]: Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck

Apr 10 06:43:45 mysql-01 /etc/mysql/debian-start[21810]: This installation of MySQL is already upgraded to 5.5.29, use --force if you still need to run mysql_upgrade

Apr 10 06:43:45 mysql-01 /etc/mysql/debian-start[21821]: Checking for insecure root accounts. Apr 10 06:43:45 mysql-01 /etc/mysql/debian-start[21826]: Triggering myisam-recover for all MyISAM tables

Any help diagnosing this would be much appreciated!

share|improve this question
    
512 MB, seriously? –  NickW Apr 10 '13 at 11:17
    
Yes, seriously. –  robmcvey Apr 10 '13 at 11:27
1  
That's a really small amount of memory to run linux on nowadays, let alone a mysql server. As for diagnosing, I'd look into things like log rotation, cron and update scripts for apt.. –  NickW Apr 10 '13 at 11:32
2  
Nick with all due respect, unless you know exactly what this database is used for and what the performance requirements are, I can't see why you are commenting about the amount of RAM. –  robmcvey Apr 10 '13 at 11:41
    
What uses the database? A program on another server? How many connections are allowed at once? –  DerfK Apr 10 '13 at 11:53
show 5 more comments

1 Answer

The kernel is detecting that it is running out of memory, possibly because some process is running wild.

Usually OOM killer will try to identify this process, and kill it. The reason it is killing mysql is because this is probably the process that is currently taking the most amount of ram, so it's a very likely candidate for the running wild process.

However, it also seems like snmpd is the culprit. (it is taking 160MB's which is a lot) snmpd is a deamon responsible for listening for snmp traffic, it seems weird for it to take this much memory.

Since this is happening each day at the same time, check your daily cron jobs. And check your snmpd log file. Also check for incomming connections around that time. (from sshd)

All these log files should be showing up somewhere in /var/log/xxx

If this turns up nothing unexpected, look in the log files for the other processes mentioned in the log. (mysql and rsyslogd)

Also, from your graph you only have 66MB free on average, and are running in to memory issues way more then just at 6.40, almost 20% of the time you seem to have less then a few MB's free, never more then 100MB free. (if I correctly see that the magenta bar is the free memory?)

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.