2

So a little background, a few weeks ago I brought a new CentOS dedicated server in Melbourne, however it seems beside the number of attackers I get, there also seem to be an issue with the MySQL database, or a software that is using is as it keep crashing and drying.

I have gone thru the logs and don't see any reason why it would be crashing but i am wondering if anyone out there would be able to give me a hand fixing this issue.

Last part of the log file:

 150512 06:15:02 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
150512 06:20:02 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150512  6:20:02 [Warning] Using unique option prefix key_buffer instead of key_buffer_size is deprecated and will be removed in a future release. Please use the full name instead.
150512  6:20:02 [Note] Plugin 'FEDERATED' is disabled.
150512  6:20:02 InnoDB: The InnoDB memory heap is disabled
150512  6:20:02 InnoDB: Mutexes and rw_locks use GCC atomic builtins
150512  6:20:02 InnoDB: Compressed tables use zlib 1.2.3
150512  6:20:02 InnoDB: Using Linux native AIO
150512  6:20:02 InnoDB: Initializing buffer pool, size = 128.0M
150512  6:20:02 InnoDB: Completed initialization of buffer pool
150512  6:20:02 InnoDB: highest supported file format is Barracuda.
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
150512  6:20:02  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
150512 06:20:02 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
150512 06:25:02 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150512  6:25:02 [Warning] Using unique option prefix key_buffer instead of key_buffer_size is deprecated and will be removed in a future release. Please use the full name instead.
150512  6:25:02 [Note] Plugin 'FEDERATED' is disabled.
150512  6:25:02 InnoDB: The InnoDB memory heap is disabled
150512  6:25:02 InnoDB: Mutexes and rw_locks use GCC atomic builtins
150512  6:25:02 InnoDB: Compressed tables use zlib 1.2.3
150512  6:25:02 InnoDB: Using Linux native AIO
150512  6:25:02 InnoDB: Initializing buffer pool, size = 128.0M
150512  6:25:02 InnoDB: Completed initialization of buffer pool
150512  6:25:02 InnoDB: highest supported file format is Barracuda.
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
150512  6:25:02  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
150512 06:25:03 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
150512 06:30:01 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150512  6:30:01 [Warning] Using unique option prefix key_buffer instead of key_buffer_size is deprecated and will be removed in a future release. Please use the full name instead.
150512  6:30:01 [Note] Plugin 'FEDERATED' is disabled.
150512  6:30:01 InnoDB: The InnoDB memory heap is disabled
150512  6:30:01 InnoDB: Mutexes and rw_locks use GCC atomic builtins
150512  6:30:01 InnoDB: Compressed tables use zlib 1.2.3
150512  6:30:01 InnoDB: Using Linux native AIO
150512  6:30:01 InnoDB: Initializing buffer pool, size = 128.0M
150512  6:30:01 InnoDB: Completed initialization of buffer pool
150512  6:30:01 InnoDB: highest supported file format is Barracuda.
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
150512  6:30:01  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
150512  6:30:01  InnoDB: Waiting for the background threads to start
150512  6:30:02 InnoDB: 5.5.41 started; log sequence number 68534366
150512  6:30:02 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3306
150512  6:30:02 [Note]   - '0.0.0.0' resolves to '0.0.0.0';
150512  6:30:02 [Note] Server socket created on IP: '0.0.0.0'.
150512  6:30:02 [Note] Event Scheduler: Loaded 0 events
150512  6:30:02 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.41'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Community Server (GPL) by Remi
150512 06:42:37 mysqld_safe Number of processes running now: 0
150512 06:42:37 mysqld_safe mysqld restarted
150512  6:42:37 [Warning] Using unique option prefix key_buffer instead of key_buffer_size is deprecated and will be removed in a future release. Please use the full name instead.
150512  6:42:37 [Note] Plugin 'FEDERATED' is disabled.
150512  6:42:37 InnoDB: The InnoDB memory heap is disabled
150512  6:42:37 InnoDB: Mutexes and rw_locks use GCC atomic builtins
150512  6:42:37 InnoDB: Compressed tables use zlib 1.2.3
150512  6:42:37 InnoDB: Using Linux native AIO
150512  6:42:37 InnoDB: Initializing buffer pool, size = 128.0M
150512  6:42:37 InnoDB: Completed initialization of buffer pool
150512  6:42:37 InnoDB: highest supported file format is Barracuda.
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
150512  6:42:37  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
150512  6:42:37  InnoDB: Waiting for the background threads to start
150512 06:42:37 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
150512 06:45:02 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150512  6:45:02 [Warning] Using unique option prefix key_buffer instead of key_buffer_size is deprecated and will be removed in a future release. Please use the full name instead.
150512  6:45:02 [Note] Plugin 'FEDERATED' is disabled.
150512  6:45:02 InnoDB: The InnoDB memory heap is disabled
150512  6:45:02 InnoDB: Mutexes and rw_locks use GCC atomic builtins
150512  6:45:02 InnoDB: Compressed tables use zlib 1.2.3
150512  6:45:02 InnoDB: Using Linux native AIO
150512  6:45:02 InnoDB: Initializing buffer pool, size = 128.0M
150512  6:45:02 InnoDB: Completed initialization of buffer pool
150512  6:45:02 InnoDB: highest supported file format is Barracuda.
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
150512  6:45:02  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
150512  6:45:02  InnoDB: Waiting for the background threads to start
150512  6:45:03 InnoDB: 5.5.41 started; log sequence number 68578838
150512  6:45:03 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3306
150512  6:45:03 [Note]   - '0.0.0.0' resolves to '0.0.0.0';
150512  6:45:03 [Note] Server socket created on IP: '0.0.0.0'.
150512  6:45:03 [Note] Event Scheduler: Loaded 0 events
150512  6:45:03 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.41'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL \

The server is running the following:

CentOS 6 vestaCP MYSQL 5.5.4 Mail Server

3 Answers 3

0
InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery
  1. Whenever database get crashed, we used to get this. Innodb mechanism tries to recover spontaneously. Use innodb_force_recovery to force the InnoDB storage engine to start up

  2. If this is happening regularly at the same time check for any cron or any logrotation (/etc/logrotate.d/mysql) getting triggered.

  3. Check if you are running out of memory. If Apache or other process also running in the same server you probably need more memory. Check the situation of your swap

    cat /proc/swaps

    Consider to add swap in that case


  • If you restart MySQL with innodb_force_recovery then dump the corrupted databases.

    mysqldump -u root -p –all-databases > all_dbs.sql

  • After dump, shutdown MySQL, move the ib* files from /var/lib/mysql/ directory.

    mkdir /var/lib/ib_files/ mv /var/lib/mysql/ib* /var/lib/ib_files/

  • Then remove ‘innodb_force_recovery’ from /etc/my.cnf and start MySQL . Check the mysqld.log if any error. Once you have a clean MySQL start, restore the databases from the dump

    mysql -u root -p < all_dbs.sql

  • Once the restoration is completed, run a database repair to simply ensure that everything is intact:

    mysqlcheck –all-databases –repair

  • After repair, restart mysql once again

    service mysql restart

0

It looks like your logs are corrupted. Try recreating the InnoDB log files by moving the existing ib_logfile(s) out of the way and restarting MySQL. If that doesn't work you're likely to have some data corruption.

0

This is an old question, but the answer can be useful to other...

One common cause of "sudden" MySQL/MariaDB crashes is the kernel OOM killer - basically, if the machine is short on RAM, the kernel kill the most RAM-consuming process.

The kernel OOM killer is partially configurable, so one should be able to "protect" an important process. However, the real solution is to understand why the machine faces an out-of-memory condition; the most common cause:

  • no or insufficient swap space configured;
  • a memory leak in some application;
  • a badly configured database (es: a too big InnoDB buffer);
  • too little memory for the actual workload.

Not the answer you're looking for? Browse other questions tagged .