Last days mysql server on my vps start crashing, sometimes right after restart.
I found that problem in wordpress website. After
service mysql start many apache2 processes overload memory and cpu. This is screenshot of top.
I didn't find anything usefull from apache logs, mysql error log show messages about lack of memory:
140922 12:24:43 [Note] Plugin 'FEDERATED' is disabled. 140922 12:24:43 InnoDB: The InnoDB memory heap is disabled 140922 12:24:43 InnoDB: Mutexes and rw_locks use GCC atomic builtins 140922 12:24:43 InnoDB: Compressed tables use zlib 1.2.8 140922 12:24:43 InnoDB: Using Linux native AIO 140922 12:24:43 InnoDB: Initializing buffer pool, size = 128.0M InnoDB: mmap(137363456 bytes) failed; errno 12 140922 12:24:43 InnoDB: Completed initialization of buffer pool 140922 12:24:43 InnoDB: Fatal error: cannot allocate memory for the buffer pool 140922 12:24:43 [ERROR] Plugin 'InnoDB' init function returned error. 140922 12:24:43 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed. 140922 12:24:43 [ERROR] Unknown/unsupported storage engine: InnoDB 140922 12:24:43 [ERROR] Aborting 140922 12:24:43 [Note] /usr/sbin/mysqld: Shutdown complete
First I thought maybe problem in db and checked database with
mysqlcheck --all-databases but all ok, I tried drop and create new database, no luck.
Also I tried to delete wordpress plugins folder. Wordpress is last version, comments disabled, 3 users, little amount of visitors (corporate blog).
Now I don't know where to look and how to diagnose problem, mysql lives no more than 10-20 sec if wordpress config enabled in apache.
UPDATE: I'm deleted WP folder and download new WP without any config. If wp site is enabled, vps starts overloading, apache can't answer to requests. Without WP enabled all works smooth. I don't destroy this droplet yet because I want found the cause of the problem. Maybe this is some exploit in vps?
UPDATE2: after searching in logs and netstat I found that problem appears when:
- wordpress site was enabled and apache with mysql works
- Some request from dummy bot for rpc.php or something like this set apache connection in CLOSE_WAIT, after several request apache had many workers in CLOSE_WAIT condition, as result many workers and luck of memory (yes I should reduce max, but anyway site stopped work when all workers waiting)
- MySQL falling with luck of memory (it's not the problem as suggested in answer it's consequence of another problems, and upgrade VM not solve the issue)
- After mysql falling overload stops. (this led me to think that perhaps the database is corrupted, but after checking db not errors were found)
What reason may enter apache in CLOSE_WAIT state after dummy request for missing file?