0

last night I was doing a server maintenance that required DB reboot. No changes in a config were made.

Current setup is as follows:

master (server1) - master (server2) and slave (server0)

MySQL version: mysql-5.5.32

server0 and server1 started fine after a reboot but when i try to start server0 it crashes with signal 11 error:

Here's the log output:

140604 03:10:08 mysqld_safe Starting mysqld daemon with databases from /opt/mysql/mysql_data
140604  3:10:08 [Note] Plugin 'FEDERATED' is disabled.
140604  3:10:08 InnoDB: The InnoDB memory heap is disabled
140604  3:10:08 InnoDB: Mutexes and rw_locks use GCC atomic builtins
140604  3:10:08 InnoDB: Compressed tables use zlib 1.2.3
140604  3:10:08 InnoDB: Using Linux native AIO
140604  3:10:08 InnoDB: Initializing buffer pool, size = 32.0G
140604  3:10:10 InnoDB: Completed initialization of buffer pool
02:10:10 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

key_buffer_size=67108864
read_buffer_size=8388608
max_used_connections=0
max_threads=2000
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 49240442 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x40000
/usr/local/mysql/bin/mysqld(my_print_stacktrace+0x35)[0x7a2825]
/usr/local/mysql/bin/mysqld(handle_fatal_signal+0x403)[0x670b43]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xf030)[0x7f1c2db0e030]
/usr/local/mysql/bin/mysqld[0x93ec38]
/usr/local/mysql/bin/mysqld[0x8328d2]
/usr/local/mysql/bin/mysqld[0x861511]
/usr/local/mysql/bin/mysqld[0x7f302f]
/usr/local/mysql/bin/mysqld[0x7c40dd]
/usr/local/mysql/bin/mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x48)[0x673948]
/usr/local/mysql/bin/mysqld[0x58307a]
/usr/local/mysql/bin/mysqld(_Z11plugin_initPiPPci+0xb5d)[0x586c9d]
/usr/local/mysql/bin/mysqld[0x507bab]
/usr/local/mysql/bin/mysqld(_Z11mysqld_mainiPPc+0x3e2)[0x508772]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xfd)[0x7f1c2ccccead]
/usr/local/mysql/bin/mysqld[0x4fe17a]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
140604 03:10:10 mysqld_safe mysqld from pid file /opt/mysql/mysql_data/odb2.pid ended

After that I started mysql with innodb_force_recovery = 1 and server started ok.

Looking at the documentation:

*1 (SRV_FORCE_IGNORE_CORRUPT) Let the server run even if it detects a corrupt page. Try to make SELECT * FROM tbl_name jump over corrupt index records and pages, which helps in dumping tables.*

So my next step was to run mysqlcheck on all databases but no errors were found.

Any ideas how to bring that server to life? I have a backup of all databases (~300GB) but restoring it would take a considerable amount of time and if possible I'd like to avoid that

2
  • It may not really matter but server0 and server1 started fine after a reboot but when i try to start server0 ... doesn't make sense
    – user9517
    Jun 4, 2014 at 8:57
  • sorry for not being clear about that: started server1, tried starting server2 (they're both masters). Tried startign server0 (slave) which booted fine.
    – ptrh
    Jun 4, 2014 at 8:58

1 Answer 1

1

Turns out it was a bug introduced in 5.5.32 http://bugs.mysql.com/bug.php?id=69623 where MySQL fails to open a second tablespace if stored in more than 1 file.

Update to 5.5.39 solved the issue

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .