2

I'm running a Wordpress site with Nginx, MariaDB, PHP-FPM and getting bombarded by a lot of different 404 request from a lot of IP (~10.000 different IP per hour requesting random URL which results in very high SQL load and random downtime).

I've tried to put the main server behind a different Nginx server which will do reverse proxy caching of the site to reducing load but the main server still get very high load because of 404 requests get passed the Nginx proxy caching server.

The server now making 5XX error because MYSQLD taking all the CPU to handle its stuffs thus make PHP-FPM starve and unresponsive to Nginx's request I think?

I get a lot of this in error log:

2017/05/13 03:48:40 [error] 24894#24894: *2936187 upstream timed out (110: Connection timed out) while connecting to upstream

My server having 16 core, 64GB RAM with 200GB SSD disk running Ubuntu 17.04 and MYSQLD always taking all the CPU as much as it could get.

My main server Nginx config:

user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;

events {
    worker_connections 2048;
}

http {
    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    keepalive_timeout 65;
    types_hash_max_size 2048;
    client_max_body_size 32M;
    disable_symlinks off;

    include /etc/nginx/mime.types;
    default_type application/octet-stream;

    gzip off;

    ### START SERVER CONFIG
    server {
        listen 80 default_server;
        root /var/www/html;

        index index.php index.html index.htm;

        access_log /var/log/nginx/access.log;
        error_log /var/log/nginx/error.log;

        server_name _;

        location / {
            try_files $uri $uri/ /index.php?$args;
        }

        location ~ \.php$ {
            include snippets/fastcgi-php.conf;
            fastcgi_pass 127.0.0.1:9000;
        }

        location ~ /\.ht {
            deny all;
        }
    }
    ### END OF SERVER CONFIG
}

PHP-FPM config:

[www]
user = www-data
group = www-data
listen = 127.0.0.1:9000
listen.owner = www-data
listen.group = www-data
listen.allowed_clients = 127.0.0.1
process.priority = -10
pm = dynamic
pm.max_children = 64
pm.start_servers = 32
pm.min_spare_servers = 2
pm.max_spare_servers = 32

Is there somehow I can improve the situation? As I said, all the request are coming from a lot of different IPs requesting a different URL with a very legit looking request (header look exactly like a browser) so I can't making any firewall rules to blocking it but I know they're automated request since there is some user-agent telling it's coming from IA64 architecture which no way any of my visitor have it.

And No, I can't use Cloudflare or similar services to prevent automated request for some reason... So is there any Nginx plugin to detect if it's a real browser load or a bot by testing javascript or similar method before allows it enter the site?

8
  • fail2ban will help prevent repeated requests from the same IPs, useful if its a set of IPs, not so much if it's a large set. Nginx page caching can also make a huge difference.
    – Tim
    May 13, 2017 at 3:16
  • The caching won't help much since the 404 requests are unique thus forcing any kind of caching to passed the request to the real server, I've told that I've setup a Nginx caching on a different server and it's doing the work, but the main server still getting so much 404 request and mostly none 200 request.
    – minhng99
    May 13, 2017 at 3:21
  • Are the requests such that you could have Nginx return a 404 based on some kind of matching before it hits PHP? PHP execution is the expensive part. Wordpress has a very specific set of URLs that users can call, but you'd need regular expressions to match your setup. There are a few more ideas here.
    – Tim
    May 13, 2017 at 3:40
  • No, the request are very legit I couldn't find anything that suspicious enough to making a pattern out of it on reading logs except the 404 return
    – minhng99
    May 13, 2017 at 4:44
  • So I'm having a idea... is there anyway I could generate html static pages on every corner of my WP and pass it to Nginx instead of all those PHP SQL processing? and 404 will get ignored without it reached PHP.
    – minhng99
    May 13, 2017 at 4:49

1 Answer 1

0

At first I'd study the incoming requests. Are they really attacks or does your application have many broken links? If you can fix the cause, it's always better.

Fail2Ban is also something I'd recommend, though won't do much if every IP does just one request.

Regardless, you will want to avoid a 404 to reach Wordpress/PHP/MySQL. If there's any pattern in the request you can match on, the webserver could handle it. If no clear pattern, it's more tricky but can still be done.

These instructions for MySQL can be adapted for Nginx:

https://www.pipeten.info/2015/10/better-handling-wordpress-404-errors/

But what may be even better is Repsheet.

https://getrepsheet.com/

It can help judge whether a request is one you want or not, and process them differently. These IPs doing random 404's clearly don't mimic normal user behaviour. Repsheet would be able to tell after some learning, and then you can dish out a 404 or 403 before it hits the full web stack.

Repsheet has a module for Nginx: https://github.com/repsheet/repsheet-nginx

The other way around, it will come to know your real (repeat) users as good actors, and you can set rules to prioritise them.

Lastly, because most HTTP bots are quite stupid, you can use the Test Cookie Module for Nginx to test whether this is a true user agent or not:

https://github.com/kyprizel/testcookie-nginx-module

(But be careful blocking good bots like Google, don't kill your SEO, whitelist those!)

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .