Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.


I am using godaddy's dedicate server to host my web application, it's centOS 5(I have no idea why they are still using this.), I installed mysql in that server, everything was running OK since one day I got an report saying that the web application cannot connect to mysql, I checked and found out that mysql crashed, after restarting mysql, everything worked again. But the weird thing is that it will crash randomly, sometimes once in a month, sometimes once or twice in couple months. Please help me figure out what the problem is. The following is the most recent crash log from mysql. Thank you very much for your time.

121015 08:59:35 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
121015  8:59:36 [Note] Plugin 'FEDERATED' is disabled.
121015  8:59:36 InnoDB: The InnoDB memory heap is disabled
121015  8:59:36 InnoDB: Mutexes and rw_locks use InnoDB's own implementation
121015  8:59:36 InnoDB: Compressed tables use zlib 1.2.3
121015  8:59:36 InnoDB: Using Linux native AIO
121015  8:59:36 InnoDB: Initializing buffer pool, size = 128.0M
121015  8:59:36 InnoDB: Completed initialization of buffer pool
121015  8:59:36 InnoDB: highest supported file format is Barracuda.
121015  8:59:38  InnoDB: Waiting for the background threads to start
121015  8:59:39 InnoDB: 1.1.8 started; log sequence number 3584614
121015  8:59:39 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3306
121015  8:59:39 [Note]   - '0.0.0.0' resolves to '0.0.0.0';
121015  8:59:39 [Note] Server socket created on IP: '0.0.0.0'.
121015  8:59:40 [Note] Event Scheduler: Loaded 2 events
121015  8:59:40 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.27'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Community Server (GPL) by Remi
121019 23:42:58 [Warning] IP address '118.186.38.55' could not be resolved: Name or service not known
121024 05:58:39 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
121024  5:58:41 [Note] Plugin 'FEDERATED' is disabled.
121024  5:58:41 InnoDB: The InnoDB memory heap is disabled
121024  5:58:41 InnoDB: Mutexes and rw_locks use InnoDB's own implementation
121024  5:58:41 InnoDB: Compressed tables use zlib 1.2.3
121024  5:58:41 InnoDB: Using Linux native AIO
121024  5:58:41 InnoDB: Initializing buffer pool, size = 128.0M
121024  5:58:41 InnoDB: Completed initialization of buffer pool
121024  5:58:41 InnoDB: highest supported file format is Barracuda.
InnoDB: The log sequence number in ibdata files does not match
nnoDB: the log sequence number in the ib_logfiles!
121024  5:58:41  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
121024  5:58:43  InnoDB: Waiting for the background threads to start
121024  5:58:44 InnoDB: 1.1.8 started; log sequence number 3584963
121024  5:58:44 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3306
121024  5:58:44 [Note]   - '0.0.0.0' resolves to '0.0.0.0';
121024  5:58:44 [Note] Server socket created on IP: '0.0.0.0'.
121024  5:58:45 [Note] Event Scheduler: Loaded 2 events
121024  5:58:45 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.27'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Community Server (GPL) by Remi
121024  6:06:32 [Note] /usr/libexec/mysqld: Normal shutdown

121024  6:06:32 [Note] Event Scheduler: Purging the queue. 2 events
121024  6:06:34 [Warning] /usr/libexec/mysqld: Forcing close of thread 3  user: 'root'

121024  6:06:34 [Warning] /usr/libexec/mysqld: Forcing close of thread 2  user: 'root'

121024  6:06:34  InnoDB: Starting shutdown...
121024  6:06:37  InnoDB: Shutdown completed; log sequence number 3584963
121024  6:06:37 [Note] /usr/libexec/mysqld: Shutdown complete
121024 06:06:37 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
121024 06:08:23 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
121024  6:08:23 [Note] Plugin 'FEDERATED' is disabled.
121024  6:08:23 InnoDB: The InnoDB memory heap is disabled
121024  6:08:23 InnoDB: Mutexes and rw_locks use InnoDB's own implementation
121024  6:08:23 InnoDB: Compressed tables use zlib 1.2.3
121024  6:08:23 InnoDB: Using Linux native AIO
121024  6:08:23 InnoDB: Initializing buffer pool, size = 128.0M
121024  6:08:23 InnoDB: Completed initialization of buffer pool
121024  6:08:23 InnoDB: highest supported file format is Barracuda.
121024  6:08:23  InnoDB: Waiting for the background threads to start
121024  6:08:24 InnoDB: 1.1.8 started; log sequence number 3584963
121024  6:08:24 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3306
121024  6:08:24 [Note]   - '0.0.0.0' resolves to '0.0.0.0';
121024  6:08:24 [Note] Server socket created on IP: '0.0.0.0'.
121024  6:08:24 [Note] Event Scheduler: Loaded 2 events
121024  6:08:24 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.27'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Community Server (GPL) by Remi
121026 10:49:09 [Warning] IP address '69.46.72.78' could not be resolved: Temporary failure in name resolution
121027  7:00:57 [Warning] IP address '119.1.96.27' could not be resolved: Name or service not known
121028  1:42:49 [Warning] IP address '211.152.54.123' could not be resolved: Name or service not known
121029 11:48:07 [Warning] IP address '58.16.63.214' could not be resolved: Name or service not known
121031 10:52:46 [Note] Event Scheduler: scheduler thread started with id 99

121105 14:04:01 [Warning] IP address '61.134.25.4' could not be resolved: Name or service not known
121107 09:18:43 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
121107  9:18:44 [Note] Plugin 'FEDERATED' is disabled.
121107  9:18:44 InnoDB: The InnoDB memory heap is disabled
121107  9:18:44 InnoDB: Mutexes and rw_locks use InnoDB's own implementation
121107  9:18:44 InnoDB: Compressed tables use zlib 1.2.3
121107  9:18:44 InnoDB: Using Linux native AIO
121107  9:18:44 InnoDB: Initializing buffer pool, size = 128.0M
121107  9:18:44 InnoDB: Completed initialization of buffer pool
121107  9:18:44 InnoDB: highest supported file format is Barracuda.
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
121107  9:18:44  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
121107  9:18:48  InnoDB: Waiting for the background threads to start
121107  9:18:49 InnoDB: 1.1.8 started; log sequence number 3609300
121107  9:18:49 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3306
121107  9:18:49 [Note]   - '0.0.0.0' resolves to '0.0.0.0';
121107  9:18:49 [Note] Server socket created on IP: '0.0.0.0'.
121107  9:18:50 [Note] Event Scheduler: Loaded 2 events
121107  9:18:50 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.27'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Community Server (GPL) by Remi
121107  9:23:34 [Note] Event Scheduler: scheduler thread started with id 5
share|improve this question

migrated from stackoverflow.com Nov 8 '12 at 2:30

This question came from our site for professional and enthusiast programmers.

    
does it crash when no InnoDB tables are being used? rephrasing, could you check to see if it crashes WHEN an InnoDB table is being used? –  Gaia Nov 7 '12 at 19:43
    
@Gaia Thank you for your reply. All tables are InnoDB except two are MyISAM in my application database. –  Jason Nov 7 '12 at 19:54

2 Answers 2

up vote 1 down vote accepted

It looks like there is an issues with InnoDB corruption. If you can get it up and running, you can try running "mysqlcheck --all-databases --quick --fast --auto-repair --force ". If that will not work, you can try to dump the content and then import everything again as this will re-create the ibdata file. If you cannot run either of these before the server restarts again, you may need to look into using the Percona Data Recovery Tool for InnoDB.

share|improve this answer
    
Thanks for your suggestion, the server is up and running after restart. I will try to use the command you mentioned. Thank you very much for your time. –  Jason Nov 7 '12 at 21:10
    
I used mysqlcheck, it reported that everything is OK? –  Jason Nov 7 '12 at 21:16
    
That message means things were fixed. If everything was fine to begin with the message would have been already current. Let the server run for a while and see if it crashes again. –  Mike_GoDaddy Nov 7 '12 at 21:28
    
OK. Thanks for your help. I will accept your answer. –  Jason Nov 7 '12 at 21:29
    
Glad you got this worked out. Let us know if you run into any more issues! –  Mike_GoDaddy Nov 7 '12 at 21:50

Check your dmesg output and look for OOM messages. If MySQL is taking up too much memory, the Out Of Memory killer will take it down.

share|improve this answer
    
Thank you very much for your reply Mark, I checked dmesg, there's nothing in there. Weird! –  Jason Nov 7 '12 at 19:58
    
Anything in the mysqld log or /var/log/messages? –  Mark Cohen Nov 7 '12 at 20:12
    
Nothing in messages, I do have the log for mysqld. There are the most recent mysqld log messages in my question. Thanks. –  Jason Nov 7 '12 at 20:23
    
I will edit it to add more messages. If you could take a look, that will be great. Thanks for your time. –  Jason Nov 7 '12 at 20:23
    
Do you have any core files in /var/lib/ for mysqld? It definitely looks like it's crashing (hence the didn't shut down cleanly message) –  Mark Cohen Nov 7 '12 at 20:40

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.