Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I am running mysql 5.5 on ubuntu 12.04
I have many databases : the main one has ~15k innodb tables with a ~2GB size and the other one is a db for simple machine forum ~50MB.

2 days ago, all my webapps were crashing because all their queries would 'Lost connection to MySQL server'.

Here is my /var/log/mysql/error.log

InnoDB: End of page dump
140108 10:42:55  InnoDB: Page checksum 3777857050, prior-to-4.0.14-form checksum 2381448317
InnoDB: stored checksum 3777857050, prior-to-4.0.14-form stored checksum 3688692048
InnoDB: Page lsn 2 764141590, low 4 bytes of lsn at page end 762375020
InnoDB: Page number (if stored to page already) 129743,
InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 0
InnoDB: Page may be an index page where index id is 32470
InnoDB: (index "PRIMARY" of table "gwc"."xxia_minute_history")
InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 129743.
InnoDB: You may have to recover from a backup.
InnoDB: It is also possible that your operating
InnoDB: system has corrupted its own file cache
InnoDB: and rebooting your computer removes the
InnoDB: error.
InnoDB: If the corrupt page is an index page
InnoDB: you can also try to fix the corruption
InnoDB: by dumping, dropping, and reimporting
InnoDB: the corrupt table. You can use CHECK
InnoDB: TABLE to scan your table for corruption.
InnoDB: See also http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
InnoDB: Ending processing because of a corrupt database page.
140108 10:42:55  InnoDB: Assertion failure in thread 2918591296 in file buf0buf.c line 3631
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.

000000000000000000                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                ;
InnoDB: End of page dump
140110 17:25:34  InnoDB: Page checksum 1575996416, prior-to-4.0.14-form checksum 1371122432
InnoDB: stored checksum 0, prior-to-4.0.14-form stored checksum 0
InnoDB: Page lsn 0 0, low 4 bytes of lsn at page end 0
InnoDB: Page number (if stored to page already) 0,
InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 0
InnoDB: Page may be a freshly allocated page
140110 17:25:34  InnoDB: Assertion failure in thread 3054144320 in file btr0btr.c line 719
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
22:25:34 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

key_buffer_size=16777216
read_buffer_size=131072
max_used_connections=1
max_threads=151
thread_count=1
connection_count=1
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 346066 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0xffffffffb953e830
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = ffffffffb60a82cc thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace+0x33)[0xb7314873]
/usr/sbin/mysqld(handle_fatal_signal+0x484)[0xb71c1454]
[0xb6e77500]
/usr/sbin/mysqld(+0x4f2041)[0xb738c041]
/usr/sbin/mysqld(+0x4f2a13)[0xb738ca13]
/usr/sbin/mysqld(+0x5338fc)[0xb73cd8fc]
/usr/sbin/mysqld(+0x533db6)[0xb73cddb6]
/usr/sbin/mysqld(+0x49cf2c)[0xb7336f2c]
/usr/sbin/mysqld(_ZN7handler7ha_openEP5TABLEPKcii+0x43)[0xb71c63a3]
/usr/sbin/mysqld(_Z21open_table_from_shareP3THDP11TABLE_SHAREPKcjjjP5TABLEb+0x65d)[0xb7119f3d]
/usr/sbin/mysqld(_Z10open_tableP3THDP10TABLE_LISTP11st_mem_rootP18Open_table_context+0xef5)[0xb704ba15]
/usr/sbin/mysqld(_Z11open_tablesP3THDPP10TABLE_LISTPjjP19Prelocking_strategy+0x5e3)[0xb704c763]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x1855)[0xb7089595]
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0xfc)[0xb709072c]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x18da)[0xb70920da]
/usr/sbin/mysqld(_Z10do_commandP3THD+0xd3)[0xb70929f3]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x1eb)[0xb7145e1b]
/usr/sbin/mysqld(handle_one_connection+0x50)[0xb7145e80]
/lib/i386-linux-gnu/libpthread.so.0(+0x6d4c)[0xb6df8d4c]
/lib/i386-linux-gnu/libc.so.6(clone+0x5e)[0xb6c08dde]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (ffffffffab213018): is an invalid pointer
Connection ID (thread ID): 2
Status: NOT_KILLED

How to pinpoint the cause of all this?
* Was it my webapp that tried to insert an incorrectly formatted row in the db?
* Perhaps that was caused by the system running out of memory?
* What about using apt-get that would mess up the libraries versions/permissions?
* Perhaps a restart that didn't properly shut down mysql?
* I always have a lot of spammers/hackers that try a bunch of stuff on my server, in the apache2 access log I see things like ?page=admin;php5; and other weird stuff

I am looking to learn some new insights on how to troubleshoot my linux server.

If it matters, I run python3 bottle as a webapp and I also have php smf running.

share|improve this question
    
To troubleshoot it, first read what it says. I mean "Database page corruption on disk or a failed file read" from the log. Check your disks for SMART errors, and try to recover your tables as suggested in the log. –  Andrey Sapegin Feb 2 at 20:21
    
Already recovered the tables. Know the error is page corruption. Want to know what caused it. –  Lazik Feb 2 at 21:35

1 Answer 1

Well, file read error could be caused by problems with hard drives or filesystem. Have you tried to check your drives for errors using SMART or any other tools? Try also checking your filesystem.

Sometimes, if your server is overloaded, it could cause timeout violation on healthy hard drive as well.

Another reason, as also mentioned there in the log ("This could be because you hit a bug."), is a bug in MySQL. I just googled for it and found the following page: http://bugs.mysql.com/bug.php?id=69476. To avoid it, you could try to switch from InnoDB to MyISAM.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.