I am running mysql 5.5 on ubuntu 12.04
I have many databases : the main one has ~15k innodb tables with a ~2GB size and the other one is a db for simple machine forum ~50MB.
2 days ago, all my webapps were crashing because all their queries would 'Lost connection to MySQL server'.
Here is my /var/log/mysql/error.log
InnoDB: End of page dump 140108 10:42:55 InnoDB: Page checksum 3777857050, prior-to-4.0.14-form checksum 2381448317 InnoDB: stored checksum 3777857050, prior-to-4.0.14-form stored checksum 3688692048 InnoDB: Page lsn 2 764141590, low 4 bytes of lsn at page end 762375020 InnoDB: Page number (if stored to page already) 129743, InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 0 InnoDB: Page may be an index page where index id is 32470 InnoDB: (index "PRIMARY" of table "gwc"."xxia_minute_history") InnoDB: Database page corruption on disk or a failed InnoDB: file read of page 129743. InnoDB: You may have to recover from a backup. InnoDB: It is also possible that your operating InnoDB: system has corrupted its own file cache InnoDB: and rebooting your computer removes the InnoDB: error. InnoDB: If the corrupt page is an index page InnoDB: you can also try to fix the corruption InnoDB: by dumping, dropping, and reimporting InnoDB: the corrupt table. You can use CHECK InnoDB: TABLE to scan your table for corruption. InnoDB: See also http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. InnoDB: Ending processing because of a corrupt database page. 140108 10:42:55 InnoDB: Assertion failure in thread 2918591296 in file buf0buf.c line 3631 InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 000000000000000000 ; InnoDB: End of page dump 140110 17:25:34 InnoDB: Page checksum 1575996416, prior-to-4.0.14-form checksum 1371122432 InnoDB: stored checksum 0, prior-to-4.0.14-form stored checksum 0 InnoDB: Page lsn 0 0, low 4 bytes of lsn at page end 0 InnoDB: Page number (if stored to page already) 0, InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 0 InnoDB: Page may be a freshly allocated page 140110 17:25:34 InnoDB: Assertion failure in thread 3054144320 in file btr0btr.c line 719 InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 22:25:34 UTC - mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. key_buffer_size=16777216 read_buffer_size=131072 max_used_connections=1 max_threads=151 thread_count=1 connection_count=1 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 346066 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0xffffffffb953e830 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = ffffffffb60a82cc thread_stack 0x30000 /usr/sbin/mysqld(my_print_stacktrace+0x33)[0xb7314873] /usr/sbin/mysqld(handle_fatal_signal+0x484)[0xb71c1454] [0xb6e77500] /usr/sbin/mysqld(+0x4f2041)[0xb738c041] /usr/sbin/mysqld(+0x4f2a13)[0xb738ca13] /usr/sbin/mysqld(+0x5338fc)[0xb73cd8fc] /usr/sbin/mysqld(+0x533db6)[0xb73cddb6] /usr/sbin/mysqld(+0x49cf2c)[0xb7336f2c] /usr/sbin/mysqld(_ZN7handler7ha_openEP5TABLEPKcii+0x43)[0xb71c63a3] /usr/sbin/mysqld(_Z21open_table_from_shareP3THDP11TABLE_SHAREPKcjjjP5TABLEb+0x65d)[0xb7119f3d] /usr/sbin/mysqld(_Z10open_tableP3THDP10TABLE_LISTP11st_mem_rootP18Open_table_context+0xef5)[0xb704ba15] /usr/sbin/mysqld(_Z11open_tablesP3THDPP10TABLE_LISTPjjP19Prelocking_strategy+0x5e3)[0xb704c763] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x1855)[0xb7089595] /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0xfc)[0xb709072c] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x18da)[0xb70920da] /usr/sbin/mysqld(_Z10do_commandP3THD+0xd3)[0xb70929f3] /usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x1eb)[0xb7145e1b] /usr/sbin/mysqld(handle_one_connection+0x50)[0xb7145e80] /lib/i386-linux-gnu/libpthread.so.0(+0x6d4c)[0xb6df8d4c] /lib/i386-linux-gnu/libc.so.6(clone+0x5e)[0xb6c08dde] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (ffffffffab213018): is an invalid pointer Connection ID (thread ID): 2 Status: NOT_KILLED
How to pinpoint the cause of all this?
* Was it my webapp that tried to insert an incorrectly formatted row in the db?
* Perhaps that was caused by the system running out of memory?
* What about using apt-get that would mess up the libraries versions/permissions?
* Perhaps a restart that didn't properly shut down mysql?
* I always have a lot of spammers/hackers that try a bunch of stuff on my server, in the apache2 access log I see things like ?page=admin;php5; and other weird stuff
I am looking to learn some new insights on how to troubleshoot my linux server.
If it matters, I run python3 bottle as a webapp and I also have php smf running.