Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am very new at admining mysql, and bad for me, something caused the db to get clobbered. There are many error messages in the log that I am not sure how to safely proceed. Can you give some tips?

Here's the log:

110107 15:07:15  mysqld started
110107 15:07:15  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
110107 15:07:15  InnoDB: Starting log scan based on checkpoint at
InnoDB: log sequence number 35 515914826.
InnoDB: Doing recovery: scanned up to log sequence number 35 515915839
InnoDB: 1 transaction(s) which must be rolled back or cleaned up
InnoDB: in total 1 row operations to undo
InnoDB: Trx id counter is 0 1697553664
110107 15:07:15  InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44
45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73
74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 
InnoDB: Apply batch completed
InnoDB: Starting rollback of uncommitted transactions
InnoDB: Rolling back trx with id 0 1697553198, 1 rows to undoInnoDB: Error: trying to 
access page number 3522914176 in space 0,
InnoDB: space name ./ibdata1,
InnoDB: which is outside the tablespace bounds.
InnoDB: Byte offset 0, len 16384, i/o type 10
110107 15:07:15InnoDB: Assertion failure in thread 3086403264 in file fil0fil.c line 
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: about forcing recovery.
mysqld got signal 11;

This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 217599 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0xbffc55ac, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
New value of fp=(nil) failed sanity check, terminating stack trace!

 Please read and follow 
instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do 
resolve it The manual page at contains
information that should help you find out what is causing the crash.
110107 15:07:15  mysqld ended
share|improve this question
where is the error ? – silviud Jan 8 '11 at 16:13
Restore from your latest backup. You do keep those, right? – jgoldschrafe Jan 8 '11 at 16:40
the error is in the mysqld log file. And yes I have backups and they are pretty good but I wanted to get the up to the minute ones. Can you tell if there is a way for a cleverer DBA than me to fix the db, or at least salvage part of it? Can you tell which db is broken? – pitosalas Jan 8 '11 at 17:36
You should be able to restore your latest backup, and from the timestamp of that log- replay the binary log from that point up until the point it broke. This is assuming you have the binary log enabled. – Steve Jan 10 '11 at 0:56
@silvuid here --> mysqld got signal 11; – bahamat Jan 11 '11 at 6:18
up vote 0 down vote accepted

What version of MySQL are you using? It is possible that you have hit a bug in InnoDB, but it is also likely that you have hardware issues that have caused an InnoDB page to be corrupted. In that case you will need to restore from backup.

share|improve this answer
Thanks... I think it was hardware; we've gone ahead and restored from backups. Ouch! – pitosalas Jan 12 '11 at 4:14

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.