Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

A cron job automatically restarted my mysql database. What's the cause for the crash, or can you suggest how to resolve or monitor.

I would REALLY appreciate your input.

    120715 14:38:58  mysqld started
120715 14:38:58  InnoDB: Started; log sequence number 0 411137570
120715 14:38:58 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.95'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
120715 15:14:21 [Note] /usr/libexec/mysqld: Normal shutdown

120715 15:14:23  InnoDB: Starting shutdown...
120715 15:14:25  InnoDB: Shutdown completed; log sequence number 0 411166467
120715 15:14:25 [Note] /usr/libexec/mysqld: Shutdown complete

120715 08:14:25  mysqld ended

120715 08:14:26  mysqld started
120715  8:14:26  InnoDB: Started; log sequence number 0 411166467
120715  8:14:26 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.95'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
121212 09:15:32  mysqld started
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
121212  9:15:58  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
121212  9:17:28  InnoDB: Started; log sequence number 0 554145193
121212  9:17:57 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.95'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
share|improve this question
    
(1) There is a big jump between 120715 and 121212, Are those line numbers? (2) What happen between 8:14:26 and 9:15:32? There is no mysql crash line or shutdown line. Did the machine crashed? –  John Siu Dec 15 '12 at 18:31
    
I'm not sure what happened or what those numbers mean. Machine seems to be fine. app seems to be (create, update, delete, read data). Any other thoughts? –  user1322092 Dec 15 '12 at 18:34
    
(1) Is this an in-house box or vps? (2) How you obtain the log? (3) I would suggest nagios or collectd for monitoring mysql. (4) Is this happenning regualrly or first time? –  John Siu Dec 15 '12 at 18:35
    
Happened for the first time (in 2 and half years). I'm on a RackSpace cloud server with 512MB ram. Would 'crash recovery' have attempted to repair corrupted tables.. perhaps why app is now working. please advise –  user1322092 Dec 15 '12 at 18:38
    
The mysql auto recovered itself already else it will refuse to start. Base on that log, 120715, 121212 are actually date. I suspect you have a server crash. Check your box server log, or even give Rackspace a call or email for confirmation for server reboot happen around Dec 12, 2012, 9AM, server time. –  John Siu Dec 15 '12 at 18:42

1 Answer 1

up vote 0 down vote accepted

Base on that log, 120715, 121212 are actually server date.

On Dec 12, 2012 09:15:32 server time

121212 09:15:32  mysqld started

Mysql is started and auto recovered itself.

I suspect you have a server crash. Check your box server log, or even give Rackspace a call or email for confirmation for server reboot happen around Dec 12, 2012, 9AM, server time.

If you want some daily report of various services up and down, common choices are as follow

  1. logwatch Usually configured to give daily report of services running on a system.
  2. Nagios + nrpe A bit more involve in configuration but can give close to instant alert of service/daemon outage.
share|improve this answer
1  
Yes, the server did crash that day and time.. rather Rackspace virtualization layer crashed. –  user1322092 Dec 15 '12 at 19:16
    
I updated the answer with pointers for server monitoring. –  John Siu Dec 15 '12 at 19:18
    
Thank you John!! Actually, the very last line of my log file that would follow what I pasted above does show this message 'A mysqld process already exists at Sat Dec 15 06:20:56 PST 2012' SO, my email wasn't queue and was sent at the correct time; rather, my DB up cron job might have a bug or something... strange, except the script has been running the last few years (every 5 min). Perhaps maybe it was just a fleeting system quirk. –  user1322092 Dec 15 '12 at 19:41
    
That is very strange, but maybe related to pid file. In general mysql will refused to start if there is one running, base on pid file lock. It is possible the file system read-only mode confused it. If possible, reboot the server to clear up things. –  John Siu Dec 15 '12 at 19:47
    
Yep, can't figure out why. I use ping to test the connection. Perhaps it failed or timed out.. Thanks for your other help. I should have picked up on the date and times in the log file entries. –  user1322092 Dec 15 '12 at 20:21

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.