Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm running a web server off of an Amazon EC2 instance, and I'm using the Amazon Linux AMI as my instance operating system (I think it's CentOS or something).

Anyways for the past month or so something weird has been happening with MySQL. Around once a week, it seems that MySQL will randomly just shut off and I'll get the error "Can't connect to local MySQL server through socket mysql.sock."

The only way I've managed to fix this error is to reboot the EC2 instance, which means pretty much once a week I'm manually rebooting the instance. I'm wondering why MySQL will just randomly stop working about once a week.

Any help would be greatly appreciated.

Contents of /var/log/mysqld.log:

30414 20:03:45 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
130414 20:03:45 [Note] Plugin 'FEDERATED' is disabled.
130414 20:03:45 InnoDB: The InnoDB memory heap is disabled
130414 20:03:45 InnoDB: Mutexes and rw_locks use GCC atomic builtins
130414 20:03:45 InnoDB: Compressed tables use zlib 1.2.5
130414 20:03:45 InnoDB: Using Linux native AIO
130414 20:03:45 InnoDB: Initializing buffer pool, size = 128.0M
130414 20:03:45 InnoDB: Completed initialization of buffer pool
InnoDB: The first specified data file ./ibdata1 did not exist:
InnoDB: a new database to be created!
130414 20:03:45  InnoDB: Setting file ./ibdata1 size to 10 MB
InnoDB: Database physically writes the file full: wait...
130414 20:03:46  InnoDB: Log file ./ib_logfile0 did not exist: new to be created
InnoDB: Setting log file ./ib_logfile0 size to 5 MB
InnoDB: Database physically writes the file full: wait...
130414 20:03:46  InnoDB: Log file ./ib_logfile1 did not exist: new to be created
InnoDB: Setting log file ./ib_logfile1 size to 5 MB
InnoDB: Database physically writes the file full: wait...
InnoDB: Doublewrite buffer not found: creating new
InnoDB: Doublewrite buffer created
InnoDB: 127 rollback segment(s) active.
InnoDB: Creating foreign key constraint system tables
InnoDB: Foreign key constraint system tables created
130414 20:03:47  InnoDB: Waiting for the background threads to start
130414 20:03:48 InnoDB: 5.5.30 started; log sequence number 0
130414 20:03:48 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3306
130414 20:03:48 [Note]   - '0.0.0.0' resolves to '0.0.0.0';
130414 20:03:48 [Note] Server socket created on IP: '0.0.0.0'.
130414 20:03:48 [Note] Event Scheduler: Loaded 0 events
130414 20:03:48 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.30'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Community Server (GPL)
130519 22:55:03 [Note] /usr/libexec/mysqld: Normal shutdown

130519 22:55:03 [Note] Event Scheduler: Purging the queue. 0 events
130519 22:55:03  InnoDB: Starting shutdown...
130519 22:55:04  InnoDB: Shutdown completed; log sequence number 1595675
130519 22:55:04 [Note] /usr/libexec/mysqld: Shutdown complete

130519 22:55:04 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
130519 22:55:05 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
130519 22:55:05 [Note] Plugin 'FEDERATED' is disabled.
130519 22:55:05 InnoDB: The InnoDB memory heap is disabled
130519 22:55:05 InnoDB: Mutexes and rw_locks use GCC atomic builtins
130519 22:55:05 InnoDB: Compressed tables use zlib 1.2.5
130519 22:55:05 InnoDB: Using Linux native AIO
130519 22:55:05 InnoDB: Initializing buffer pool, size = 128.0M
130519 22:55:05 InnoDB: Completed initialization of buffer pool
130519 22:55:05 InnoDB: highest supported file format is Barracuda.
130519 22:55:06  InnoDB: Waiting for the background threads to start
130519 22:55:07 InnoDB: 5.5.31 started; log sequence number 1595675
130519 22:55:07 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3306
130519 22:55:07 [Note]   - '0.0.0.0' resolves to '0.0.0.0';
130519 22:55:07 [Note] Server socket created on IP: '0.0.0.0'.
130519 22:55:07 [Note] Event Scheduler: Loaded 0 events
130519 22:55:07 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.31'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Community Server (GPL)
130602 19:10:33 mysqld_safe Number of processes running now: 0
130602 19:10:33 mysqld_safe mysqld restarted
130602 19:10:33 [Note] Plugin 'FEDERATED' is disabled.
130602 19:10:33 InnoDB: The InnoDB memory heap is disabled
130602 19:10:33 InnoDB: Mutexes and rw_locks use GCC atomic builtins
130602 19:10:33 InnoDB: Compressed tables use zlib 1.2.5
130602 19:10:33 InnoDB: Using Linux native AIO
130602 19:10:33 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137363456 bytes) failed; errno 12
130602 19:10:33 InnoDB: Completed initialization of buffer pool
130602 19:10:33 InnoDB: Fatal error: cannot allocate memory for the buffer pool
130602 19:10:33 [ERROR] Plugin 'InnoDB' init function returned error.
130602 19:10:33 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
130602 19:10:33 [ERROR] Unknown/unsupported storage engine: InnoDB
130602 19:10:33 [ERROR] Aborting

130602 19:10:33 [Note] /usr/libexec/mysqld: Shutdown complete

130602 19:10:33 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
130602 19:15:23 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
130602 19:15:23 [Note] Plugin 'FEDERATED' is disabled.
130602 19:15:23 InnoDB: The InnoDB memory heap is disabled
130602 19:15:23 InnoDB: Mutexes and rw_locks use GCC atomic builtins
130602 19:15:23 InnoDB: Compressed tables use zlib 1.2.5
130602 19:15:23 InnoDB: Using Linux native AIO
130602 19:15:23 InnoDB: Initializing buffer pool, size = 128.0M
130602 19:15:23 InnoDB: Completed initialization of buffer pool
130602 19:15:24 InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 129807478
130602 19:15:24  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 129809289
130602 19:15:24  InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
130602 19:15:24  InnoDB: Waiting for the background threads to start
130602 19:15:25 InnoDB: 5.5.31 started; log sequence number 129809289
130602 19:15:25 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3306
130602 19:15:25 [Note]   - '0.0.0.0' resolves to '0.0.0.0';
130602 19:15:25 [Note] Server socket created on IP: '0.0.0.0'.
130602 19:15:25 [Note] Event Scheduler: Loaded 0 events
130602 19:15:25 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.31'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Community Server (GPL)
130604 12:07:41 mysqld_safe Number of processes running now: 0
130604 12:07:41 mysqld_safe mysqld restarted
130604 12:07:41 [Note] Plugin 'FEDERATED' is disabled.
130604 12:07:41 InnoDB: The InnoDB memory heap is disabled
130604 12:07:41 InnoDB: Mutexes and rw_locks use GCC atomic builtins
130604 12:07:41 InnoDB: Compressed tables use zlib 1.2.5
130604 12:07:41 InnoDB: Using Linux native AIO
130604 12:07:41 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137363456 bytes) failed; errno 12
130604 12:07:41 InnoDB: Completed initialization of buffer pool
130604 12:07:41 InnoDB: Fatal error: cannot allocate memory for the buffer pool
130604 12:07:41 [ERROR] Plugin 'InnoDB' init function returned error.
130604 12:07:41 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
130604 12:07:41 [ERROR] Unknown/unsupported storage engine: InnoDB
130604 12:07:41 [ERROR] Aborting

130604 12:07:41 [Note] /usr/libexec/mysqld: Shutdown complete

130604 12:07:41 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
130604 12:10:46 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
130604 12:10:47 [Note] Plugin 'FEDERATED' is disabled.
130604 12:10:47 InnoDB: The InnoDB memory heap is disabled
130604 12:10:47 InnoDB: Mutexes and rw_locks use GCC atomic builtins
130604 12:10:47 InnoDB: Compressed tables use zlib 1.2.5
130604 12:10:47 InnoDB: Using Linux native AIO
130604 12:10:47 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137363456 bytes) failed; errno 12
130604 12:10:47 InnoDB: Completed initialization of buffer pool
130604 12:10:47 InnoDB: Fatal error: cannot allocate memory for the buffer pool
130604 12:10:47 [ERROR] Plugin 'InnoDB' init function returned error.
130604 12:10:47 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
130604 12:10:47 [ERROR] Unknown/unsupported storage engine: InnoDB
130604 12:10:47 [ERROR] Aborting

130604 12:10:47 [Note] /usr/libexec/mysqld: Shutdown complete

130604 12:10:47 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
130604 12:10:59 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
130604 12:10:59 [Note] Plugin 'FEDERATED' is disabled.
130604 12:10:59 InnoDB: The InnoDB memory heap is disabled
130604 12:10:59 InnoDB: Mutexes and rw_locks use GCC atomic builtins
130604 12:10:59 InnoDB: Compressed tables use zlib 1.2.5
130604 12:10:59 InnoDB: Using Linux native AIO
130604 12:10:59 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137363456 bytes) failed; errno 12
130604 12:10:59 InnoDB: Completed initialization of buffer pool
130604 12:10:59 InnoDB: Fatal error: cannot allocate memory for the buffer pool
130604 12:10:59 [ERROR] Plugin 'InnoDB' init function returned error.
130604 12:10:59 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
130604 12:10:59 [ERROR] Unknown/unsupported storage engine: InnoDB
130604 12:10:59 [ERROR] Aborting
share|improve this question
    
Sorry, but if you can't tell which operating system you are using, you need a system administrator –  dawud Jun 16 '13 at 21:06
    
Sorry but that really has nothing to do with my original question. I chose the Amazon Linux AMI, it uses the yum updater and I believe it is CentOS. Instead of criticizing me just because I'm unsure of what OS I'm running, maybe you should say something that can lead to a solution. –  goddfree Jun 16 '13 at 21:25
    
Anything in the mysql error logs? The socket error is very vague and won't help narrow anything down. Running out of RAM? What else is on the box? –  Nathan C Jun 16 '13 at 21:59
    
The only thing I'm using the instance for is a web server. I'm not sure where I can get mysql error logs, but I went to /var/log/mysqld.log and copied the stuff from there into my updated post. –  goddfree Jun 16 '13 at 22:16
1  
a) Amazon Linux is based derived from CentOS 6/RHEL. b) You are likely running out of memory (InnoDB: Fatal error: cannot allocate memory for the buffer pool). Either allocate some swap space or use an instance with more memory. (Check your used memory with free and/or view your historical memory logs (use sar -r (needs the sysstat package)) –  cyberx86 Jun 17 '13 at 2:31
show 2 more comments

1 Answer

Check what is the ownership and the permissions of your /var/lib/mysql/mysql.sock. Try to check if the file exists too when you get this error message. If not, create it and restart your service to see if it works.

My guess is your VM is running out of memory. I would advise to use a lightweight webserver like Nginx, and disable all the backends you don't use in MySQL. I know that MariaDB can consume a lot of memory because of all the backends it uses.

share|improve this answer
    
Thanks, someone else also mentioned that it's probably because I'm running out of memory, so that clears up what the issue is. I'm not sure what Nginx is, but I'm just setting up my own web server by installing vsftpd, mysql, php, apache, amongst other things. I use phpmyadmin for database management, I'm not exactly sure what you mean by "backends" but really the only things I have installed are the components needed to run a web server + phpmyadmin. –  goddfree Jun 18 '13 at 12:45
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.