1

I'd like to preface this by saying I have read at least 10 related Serverfault questions before resorting to making my own...

I am currently running a Ubuntu 14.04.3 server with 2GB of RAM and about 5 active WordPress installations, all managed under the Vesta CP control panel.

Normally, it uses up about 700MB of the 2GB. But every week or so, all of the RAM becomes magically consumed, and the server slows down to almost a halt.

If I SSH into it and restart apache, as well as clear the memory (echo 3 > /proc/sys/vm/drop_caches), it starts functioning again just fine.

Here's my prefork module settings, which I feel are very reasonable:

<IfModule mpm_prefork_module>
    StartServers          5
    MinSpareServers       1
    MaxSpareServers       5
    ServerLimit          10
    MaxClients           10
    MaxRequestsPerChild  1000
</IfModule>

I even enabled mod_status and tried to see what PHP files were taking too long, but didn't find anything suspicious. Of course, when I look at the log while the server is down, it is flooded with at least 200 PHP files because they are unable to run due to the massive memory consumption.

I even enabled an 8GB SWAP file but that seems to have just delayed the inevitable.

Here's what the free -m command pulls up every time:

root@apache2-ps7881:/home/dhc-user# free -m
             total       used       free     shared    buffers     cached
Mem:          2001       1943         57         35          1         59
-/+ buffers/cache:       1883        118
Swap:         8191       4083       4108

After restarting apache:

root@apache2-ps7881:/etc/apache2# free -m
             total       used       free     shared    buffers     cached
Mem:          2001        744       1257         65         36        204
-/+ buffers/cache:        503       1498
Swap:         8191        140       8051

Here's the /var/log/apache2/error.log:

[Fri Feb 12 08:22:33.063204 2016] [mpm_prefork:error] [pid 2081] AH00161: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting
[Fri Feb 12 13:12:59.819680 2016] [core:warn] [pid 2081] AH00045: child process 6334 still did not exit, sending a SIGTERM

That "child process still did not exit" error goes on for hundreds of more lines.

I get the server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting message every time it goes down.

Another error.log reveals the following:

[Fri Feb 12 08:19:55.781598 2016] [:error] [pid 20686] [client 10.10.10.9:54559] PHP Warning:  mysqli_connect(): (08004/1040): Too many connections in /[censored]/$
[Fri Feb 12 08:19:55.896491 2016] [:error] [pid 20686] [client 10.10.10.9:54559] Too many connections, referer: http://[censored]

Could it be that there's a connection not being closed? But would that cause a memory leak?

Here's an example of what the graphs reveal during the crashes:

  • Can you post the output of "top" when this has happened? What do you mean "as well as clear the memory"? Memory management is automatic AFAIK. Do you have to restart any other services to fix? What do the error logs say around the time it starts? Add answers to your question. – Tim Feb 12 '16 at 20:09
  • @Tim I have updated the question with the command I use to clear the memory. I only restart Apache and run that command to fix the problem. I'm checking the error logs now. As for top, I will have to wait until it happens again. – Gray Adams Feb 12 '16 at 20:36
  • I note that the error logs says you have too many MySQL database connections and too many Apache workers. Seems your server is just getting really busy. Is it serving custom content to everyone, or is it serving a large number of pages that change relatively infrequently? – Tim Feb 13 '16 at 4:40
  • @Tom There is a radio player on the site which requires the client to check the database every 10 seconds for song information, and every 60 for show information. But I have a close function at the end of every script. – Gray Adams Feb 14 '16 at 17:45
  • To me it simply looks like a busy server that hits the resource limits set. You could try nginx to reduce memory usage, but PHP and MySQL are probably the main things taking up memory. – Tim Feb 14 '16 at 18:45
1

I had a similar issue where a Wordpress site would start using tons of resources to the point where the server became largely unresponsive. Investigating the logs, I saw a few hundred attempts to access xmlrpc.php right around the same time as the memory footprint would balloon. The functions in xmlrpc.php can be abused as a force multiplier in brute force attacks using the system.multicall method.

This article is a more articulate description of how it works: https://blog.sucuri.net/2015/10/brute-force-amplification-attacks-against-wordpress-xmlrpc.html

More importantly, here are a few mitigation strategies from that article:

Protecting Yourself

I used to recommend people block all access to xmlrpc.php, but it was breaking some plugin’s functionality (mostly JetPack). With that in mind, if you are not using JetPack or any of the other plugin that require it XML-RPC, it might be a good idea to block direct access to it altogether.

If you can’t block XML-RPC, and you are using a WAF (web application firewall), I highly recommend blocking system.multicall requests. It is barely used in the wild and will protect you against these amplification methods.

I don't use any plugins that require access to xmlrpc.php, so I modified .htaccess to deny access. Since then, no more malicious actors successfully crushing the site. Here's the code if you'd like to give that a try:

Using the text editor of your choice, modify /var/www/html/.htaccess to include:

    <Files "xmlrpc.php">
    Order Allow,Deny
    deny from all
    </Files>

Wordpress has additional guidelines for hardening access to your site found here: http://codex.wordpress.org/Hardening_WordPress

The Login Security Solution plugin for Wordpress may also help. I'd post the link, but I lack the reputation. Sorry!

  • So I just went through the logs for one of the sites, and I found 906 hits to that file in just one day, which I imagine is very irregular. Most of them were within just seconds of each other. I will block access to the file, as I don't need Jetpack. For now, I am going to mark this as the answer. If it happens again, it may be something else. – Gray Adams Feb 13 '16 at 21:50
  • It went down again even after blocking access to the file. Any ideas? – Gray Adams Feb 14 '16 at 17:47
  • Start by posting the information requested in all the comments above. – Tim Feb 14 '16 at 18:11
  • @Tim I believe I have. Did I miss something? – Gray Adams Feb 14 '16 at 18:14
0

It looks like the site simply gets busy, but you've said that's not the case. It's running out of server resources, so you need to perhaps limit the resources before it exhausts them.

I'd also look at the plugins Wordpress is using, and the widgets. Remove anything that's not 100% necessary, for a test. I've seen plugins and widgets do awful awful things to Wordpress websites. At the very least you should be using a caching plugin, W3 Total Cache is best, from memory.

If it was my server I'd change the underlying architecture of the server to use Nginx and HHVM, including page caching. I have four Wordpress installs and one moderately busy website running on an aws t2.micro, which is 10% of one Xeon core (burstable to one full core) and 1GB of memory, though I do also use the Amazon RDS (relational database service). My sites load VERY quickly. Note that this change isn't necessary, you can probably fix your config and it should work fine, but I don't know how.

I've written up a significant tutorial on setting up Wordpress on AWS with Nginx, HHVM, etc. You can read it here. You could quite easily set it all up alongside your current infrastructure, run Nginx on a different port, and even use the same webroot and database - but I'd create a read only database user while I was in testing. Once you're happy that it's working just swap the ports and you'll be running nginx, hhvm, and it should be more reliable. If not at least you might get some community support, as the configuration I suggest is fairly standard.

If your websites have many anonymous visitors you can server a large number of requests from the nginx page cache, which takes virtually no CPU, and doesn't hit PHP. My little instance can serve 1000 cached pages per second, sometimes more, not really sure what the bottleneck is but it does that on 2% CPU from the cache. It can only do 11 pages per second if it hits PHP, a massive difference which shows how slow executing code is. It'll make your site faster. It should make it faster for all visitors, HHVM is quite a bit faster than PHP5.

0

The configuration you've shown us is explicitly contradicted by your description of "at least 200 PHP files" and the output of top - which clearly contains more than 10 instances. That clearing the vfs cache appears to remove the symptoms is somewhat alarming - the only impact that should have is to slow the server down.

All these processes are blocked - and they are not working for the database. There looks to be some ugly file contention going on here too.

It looks like a combination of allowing too many connections combined with a blocking file lock. The latter might be a DOS attack, particularly if you are using the default session handler. If this is the case, it should be evident from your access logs.

There's no quick fix here, but your next step should be fixing the config so it actually works, and disabling all the WordPress plugins. These plugins vary greatly in quality.

Really we should be voting to close this as too broad.

  • You say this is too broad, but I think I've given more technical details than 90% of people do. I've given log files, config settings, and what I have done to troubleshoot. Mod_status does indeed reveal well over 10 files (ranges from 50-200) that are trying to be read when it goes under. – Gray Adams Feb 21 '16 at 18:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.