Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm a programmer and a total newbie as a sysadmin...

Today I started receiving error emails from our production site:

Incorrect key file for table 'xxx'; try to repair it

I found the server with 2 out of 4 cores pegged at 100% CPU by MySQL. Took down the site, the crons, everything, and about 10 minutes later CPU went to 0.

Trying to backup the DB (mysqldump) resulted in CPU at 100% again, for several minutes, without any real output to log file.

I tried REPAIR TABLE, got this:

| Table                      | Op     | Msg_type | Msg_text |
| db.table                   | repair | error    | Corrupt  |

So that didn't work...

I thought of creating a new table (InnoDB) and "INSERT SELECT" from this one to try and recover some records, but SELECTING from the table (SELECT * FROM table LIMIT 1) also sends the CPU to 100% for a couple minutes, and again gives me the "Incorrect key file" error...

I got this error for 3 of the 4 tables in this DB, all 3 of them use the ARCHIVE engine and have several million records. (Largest is 46M records)

This started about an hour ago, all of a sudden.

Any ideas what I should do next? I would ideally like to lose as few records as possible, but at the same time I need to get the site back up ASAP.

Any help will be greatly appreciated!
Thank you!

share|improve this question
How recent is the most recent backup, to get an idea of how many records you'd lose? – Bart Silverstrim Aug 4 '11 at 13:03
Sorry I missed this comment. I'd hope last night. My client's supposed to have arranged backups with the hosting company, but I don't know for a fact. I'm hoping to not have to count on Backups, ideally, and there may be none. – Daniel Magliola Aug 4 '11 at 13:15

MySQL had a long-standing bug with Archive tables starting to corrupt after they got bigger than 2 GB. How big is your table?

share|improve this answer
Hi, thank you for your answer. How do I check that? (NOTE: Trying to query information_schema.tables sends the CPU to 100% again) :-( – Daniel Magliola Aug 4 '11 at 13:07
In an old dev copy that I have (few weeks / months ago), largest table is 365 Mb, it can't have grown to 2Gb in the time since that dump... – Daniel Magliola Aug 4 '11 at 13:09
Never say never, check at the file system level. And is there disk space left in your DB partition? – Janne Pikkarainen Aug 4 '11 at 13:10
How can I check at the FS level? (Sorry, i'm a noob at this) There's 100Gb free according to DF – Daniel Magliola Aug 4 '11 at 13:14
See with ls command in your MySQL data directory, /var/lib/mysql or so. – Janne Pikkarainen Aug 4 '11 at 13:15

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.