Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I run a development server on AWS which has worked perfectly fine without a hitch. Recently, periodically the mysql server starts to not work. I then type in

service mysqld restart

which yields an error message

MySQL Daemon failed to start.

Now if I stop the instance and then start it, and then restart the server from the command line, everything works fine (this annoyingly forced me to re-edit my zone file due to changed ip addresses, but I've read that a reboot won't get rid of ip addresses). But are there any clues as to why this wouldn't be working?

Here are a few lines from var/log/mysqld.log that I cannot make heads or tails of.

130613 23:12:21 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
130613 23:12:21 [Note] Plugin 'FEDERATED' is disabled.
130613 23:12:21 InnoDB: The InnoDB memory heap is disabled
130613 23:12:21 InnoDB: Mutexes and rw_locks use GCC atomic builtins
130613 23:12:21 InnoDB: Compressed tables use zlib 1.2.5
130613 23:12:21 InnoDB: Using Linux native AIO
130613 23:12:21 InnoDB: Initializing buffer pool, size = 128.0M
130613 23:12:21 InnoDB: Completed initialization of buffer pool
InnoDB: The first specified data file ./ibdata1 did not exist:
InnoDB: a new database to be created!
130613 23:12:21  InnoDB: Setting file ./ibdata1 size to 10 MB
InnoDB: Database physically writes the file full: wait...
130613 23:12:22  InnoDB: Log file ./ib_logfile0 did not exist: new to be created
InnoDB: Setting log file ./ib_logfile0 size to 5 MB
InnoDB: Database physically writes the file full: wait...
130613 23:12:23  InnoDB: Log file ./ib_logfile1 did not exist: new to be created
InnoDB: Setting log file ./ib_logfile1 size to 5 MB
InnoDB: Database physically writes the file full: wait...
InnoDB: Doublewrite buffer not found: creating new
InnoDB: Doublewrite buffer created
InnoDB: 127 rollback segment(s) active.
InnoDB: Creating foreign key constraint system tables
InnoDB: Foreign key constraint system tables created
130613 23:12:24  InnoDB: Waiting for the background threads to start
130613 23:12:25 InnoDB: 5.5.31 started; log sequence number 0
130613 23:12:25 [Note] Server hostname (bind-address): ''; port: 3306
130613 23:12:25 [Note]   - '' resolves to '';
130613 23:12:25 [Note] Server socket created on IP: ''.
130613 23:12:25 [Note] Event Scheduler: Loaded 0 events
130613 23:12:25 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.31'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Community Server (GPL)
130614  6:08:52 [Note] /usr/libexec/mysqld: Normal shutdown

130614  6:08:53 [Note] Event Scheduler: Purging the queue. 0 events

I read at other posts on stackexchange that sometimes this can be due to disk size limitations. df - h says

Filesystem            Size  Used Avail Use% Mounted on
/dev/xvda1            7.9G  1.7G  6.2G  22% /
tmpfs                 298M     0  298M   0% /dev/shm

Anyone have any idea what else I should be hunting for?

share|improve this question
Have you considered RDS so you don't have to manage your own MySQL instance? – Drew Khoury Apr 27 '14 at 6:37

First, you have to backup your databases:

cp -r /var/lib/mysql /tmp/mysql_backup

Then, remove the file from /var/lib/mysql.

rm -rf ibdata1

Now restart the mysql server.

/etc/init.d/mysqld start
share|improve this answer
Thanks for your answer. Nitpicks: 1. rm -rf is overkill, do stick to plain rm for removing single files, use rm -r for directories. Add -f only if you cannot be bothered lifting the write protection from many files in the tree. 2. cp -r is OK, but you use all file metadata. cp -a (or better yet, rsync) is almost always preferable. – Felix Frank Aug 5 '14 at 10:49

If you're using micro or small instances, mysql database running websites can can put a pretty big load on your VMs memory which will end up constantly crashing your mysql server.

Consider using RDS which might be more expensive or at least ec2 t2.medium to save money and still run fine.

For IPs, you'd have to allocate your instance elastic IP which will stay with your instance till you get rid of it, you won't have to pay for the IP as long as you're using it.

share|improve this answer
Check with dmesg if MySQL was OOM killed. You can add a swap if you don't want to upgrade to bigger instance with more RAM at the expense of performance. – Cristi Aug 5 '14 at 13:13
Before suggesting a RDS it should be noted that this costs nearly $600 a month i made that mistake and thought i would share. – user1503606 May 20 at 15:54

The mysql instance gets killed if you are using t2.micro instance which has around only 1GB of memory. If you do ps -ef | grep apache, you can can see numbers of apache instances running . A high number of apache instances would force server to kill mysql when it is running low on memory.

share|improve this answer

Install monit to monitor Apache. When the load goes up, Monit restarts Apache gracefully and everything keeps on working.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.