This is one of the many random Tables that get corrupted. Any ideas why and what would be causing this?

How do I keep MySQL tables from crashing and MySQL from crashing?

Repairing USR_wp537

USR_wp537.rev_commentmeta                     OK
USR_wp537.rev_comments                        OK
USR_wp537.rev_links                           OK
USR_wp537.rev_options                         OK
USR_wp537.rev_postmeta                        OK
Error    : Incorrect key file for table './USR_wp537/rev_posts'; try to repair it
Error    : Incorrect key file for table 'rev_posts'; try to repair it
error    : Corrupt
USR_wp537.rev_term_relationships              OK
USR_wp537.rev_term_taxonomy                   OK
USR_wp537.rev_terms                           OK
USR_wp537.rev_usermeta                        OK

Eventually the only way to fix is to do a mysql> REPAIR TABLE <tbl> USE_FRM;

This is also mysql 5.5

top - 20:17:11 up 4 days, 8:57, 1 user, load average: 0.41, 0.37, 0.36 Tasks: 204 total, 1 running, 203 sleeping, 0 stopped, 0 zombie Cpu0 : 11.0%us, 0.3%sy, 0.0%ni, 88.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu1 : 5.0%us, 1.0%sy, 0.0%ni, 93.4%id, 0.0%wa, 0.0%hi, 0.7%si, 0.0%st Cpu2 : 10.4%us, 0.7%sy, 0.0%ni, 89.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu3 : 1.3%us, 0.3%sy, 0.0%ni, 98.0%id, 0.0%wa, 0.0%hi, 0.3%si, 0.0%st Mem: 16288440k total, 15292940k used, 995500k free, 1398928k buffers Swap: 8191992k total, 0k used, 8191992k free, 9351404k cached


max_connections = 500
max_user_connections = 40
key_buffer = 500M
myisam_sort_buffer_size = 32M
join_buffer_size = 256M
read_buffer_size = 2M
sort_buffer_size = 2M
read_rnd_buffer_size = 2M
table_cache = 1024
thread_cache_size = 16K
wait_timeout = 20
connect_timeout = 10
tmp_table_size = 256M
max_heap_table_size = 256M
max_allowed_packet = 160M
max_connect_errors = 10
query_cache_limit = 1M
query_cache_size = 64M
query_cache_type = 1
core-file-size = unlimited
open_files_limit = 8192

max_allowed_packet = 512M

key_buffer = 64M
sort_buffer = 64M
read_buffer = 16M
write_buffer = 16M

EDIT: Looks like MySQL for some reason did a restart?

130131 20:30:59 [ERROR] Got an error from thread_id=817995, /builddir/build/BUILD/mysql-5.5.28/mysql-5.5.28/storage/myisam/mi_write.c:223
130131 20:30:59 [ERROR] MySQL thread id 817998, OS thread handle 0x7f8589e2a700, query id 24842525 localhost ahmad_wp Waiting for table level lock
SELECT option_value FROM wp_options WHERE option_name = 'widget_momizat-posts-images' LIMIT 1
130131 20:30:59 [ERROR] MySQL thread id 817995, OS thread handle 0x7f858692f700, query id 24842524 localhost ahmad_wp update
INSERT INTO `wp_options` (`option_name`, `option_value`, `autoload`) VALUES ('rewrite_rules', 'a:91:{s:27:\"typename/([0-9]{4})/(.+)/?$\";s:30:\"index.php?typename=$matches[2]\";s:47:\"category/(.+?)/feed/(feed|rdf|rss|rss2|atom)/?$\";s:52:\"index.php?category_name=$matches[1]&feed=$matches[2]\";s:42:\"category/(.+?)/(feed|rdf|rss|rss2|atom)/?$\";s:52:\"index.php?category_name=$matches[1]&feed=$matches[2]\";s:35:\"category/(.+?)/page/?([0-9]{1,})/?$\";s:53:\"index.php?category_name=$matches[1]&paged=$matches[2]\";s:17:\"category/(.+?)/?$\";s:35:\"index.php?category_name=$matches[1]\";s:44:\"tag/([^/]+)/feed/(feed|rdf|rss|rss2|atom)/?$\";s:42:\"index.php?tag=$matches[1]&feed=$matches[2]\";s:39:\"tag/([^/]+)/(feed|rdf|rss|rss2|atom)/?$\";s:42:\"index.php?tag=$matches[1]&feed=$matches[2]\";s:32:\"tag/([^/]+)/page/?([0-9]{1,})/?$\";s:43:\"index.php?tag=$matches[1]&paged=$matches[2]\";s:14:\"tag/([^/]+)/?$\";s:25:
    130131 20:33:42 mysqld_safe Number of processes running now: 0
    130131 20:33:42 mysqld_safe mysqld restarted
    130131 20:33:42 [Note] libgovernor.so found
    130131 20:33:42 [Note] All governors functions found too
    130131 20:33:42 [ERROR] Governor not connected
    130131 20:33:42 [Note] All governors lve functions found too
    130131 20:33:42 [Note] Plugin 'FEDERATED' is disabled.
    130131 20:33:42 InnoDB: The InnoDB memory heap is disabled
    130131 20:33:42 InnoDB: Mutexes and rw_locks use GCC atomic builtins
    130131 20:33:42 InnoDB: Compressed tables use zlib 1.2.3
    130131 20:33:42 InnoDB: Using Linux native AIO
    130131 20:33:42 InnoDB: Initializing buffer pool, size = 128.0M
    130131 20:33:42 InnoDB: Completed initialization of buffer pool
    130131 20:33:42 InnoDB: highest supported file format is Barracuda.
    InnoDB: The log sequence number in ibdata files does not match
    InnoDB: the log sequence number in the ib_logfiles!
    130131 20:33:42  InnoDB: Database was not shut down normally!
    InnoDB: Starting crash recovery.
    InnoDB: Reading tablespace information from the .ibd files...
    InnoDB: Restoring possible half-written data pages from the doublewrite
    InnoDB: buffer...
    130131 20:33:42  InnoDB: Waiting for the background threads to start
    130131 20:33:43 InnoDB: 1.1.8 started; log sequence number 5726524860
    130131 20:33:43 [Note] Event Scheduler: Loaded 1 event
    130131 20:33:43 [Note] /usr/sbin/mysqld: ready for connections.
    Version: '5.5.28-cll-lve'  socket: '/var/lib/mysql/mysql.sock'  port: 0  MySQL Community Server (GPL)

EDIT: hard drives are clean

EDIT: using mcelog to check RAM

EDIT: the following was used to check for issues in messages

  902  fgrep -i seg /var/log/messages
  903  fgrep -i mce /var/log/messages
  904  tail /var/log/mysql/error.log
  906  smartctl --all /dev/sda
  907  smartctl -t short /dev/sda
  908  tail /var/log/mysql/error.log
  910  smartctl --all /dev/sda
  911  fgrep -i i/o /var/log/messages
  912  fgrep -i i/o /var/log/messages*
  913  fgrep -i sda /var/log/messages*
  914  fgrep -i sense /var/log/messages*
  915  egrep 'sda.*Error' sense /var/log/messages*
  916  egrep -h 'sda.*Error' sense /var/log/messages*
  917  egrep -h 'sda.*Error' /var/log/messages*

df /tmp
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda5              4031680     97304   3729576   3% /tmp
  • 4
    Why can't it? Drives can go bad at any time and are more prone to do so when new. Feb 1 '13 at 1:48
  • 3
    You've had three bad drives in 8 months and you're dubious about a fourth one? The more bad drives you have, the more believable further bad drives should be, yeah?
    – ceejayoz
    Feb 1 '13 at 2:11
  • 1
    Why are you not using a proper RAID array?
    – EEAA
    Feb 1 '13 at 3:50
  • 1
    if you boot off of a live cd, you can run badblocks with -n which will do a full read/write of the drive but use the existing data. It should still be valid after the run, but won't be as comprehensive of a test. FWIW, I have a office server on a mediocre power circuit and we lose a drive once a month without fail. A faulty power supply can also kill drives.
    – R. S.
    Feb 1 '13 at 4:47
  • 1
    Given your run of bad drives you need to start looking into why they are failing. Your problems run much deeper than just MySQL tables getting corrupted. If you don't fix the underlying problem then you're just going to keep getting errors and there's a better than average chance it will be worse than what you've experienced so far. Feb 1 '13 at 11:40

Contrary to popular belief, MySQL tables don't just crash for no reason. Crashes are often attributed to high load or too many rows. That's crap.

If you have crashed tables then there is an underlying problem with the server, such as bad memory or a bad hard drive. It could also be a bug in the MySQL server but that is very unlikely. If you are a customer of a web host there is nothing you can do except get your host to fix the underlying problem.

  • Don't have high load. The hard drive is new. The server is also not swapping and typically has 1GB free. Feb 1 '13 at 1:17
  • sorry, I think I wasn't clear. high load is not a cause of table corruption. just because you have a new drive and free memory doesn't mean that one of them is not faulty. you need to test them.
    – longneck
    Feb 1 '13 at 2:45
  • How would I test them? Feb 1 '13 at 2:45
  • memtest86 or memtest86+ would be the first stop for memory testing. Your distro likely has a package, but it will take a long time to run. And let it run for a few complete passes rather then just a single test or two.
    – R. S.
    Feb 1 '13 at 4:48

Do you have anything interesting inside messages log (usually /var/log/messages). Problems with disk or memory could be visible here...

When we're already talking about disks, I had to replace so many disks... (even the enterprise series are not bullet proof)

  • Datacenter ensured us the drive is fine. We are running memory tests with mcelog and so far nothing has shown up. Checked in and out of messages and have found nothing hardware related " fgrep -i sda /var/log/messages* -- fgrep -i sense /var/log/messages* -- egrep 'sda.*Error' sense /var/log/messages* -- egrep -h 'sda.*Error' sense /var/log/messages* -- egrep -h 'sda.*Error' /var/log/messages* -- fgrep -i i/o /var/log/messages* -- smartctl --all /dev/sda -- and so on but nothing. Feb 4 '13 at 19:34

We've seen this when the disk for the temporary tables runs out of space, due to poorly written joins. The temporary tables are transitory, so it's hard to catch in the act.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.