0

I have the following stack running on Ubuntu 18.08 and defined as docker-compose:

  1. instance of mariadb:10.3.20
  2. custom wordpress instance based on wordpress:5.3.0-php7.2 with installed ioncube on it
  3. custom nginx instance based on nginx:1.13 with installed nginx-amplify-agent on it

nginx's config:

user  nginx;
worker_processes  auto;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;

events {
    worker_connections  10000;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                          '$status $body_bytes_sent "$http_referer" '
                          '"$http_user_agent" "$http_x_forwarded_for"';

    log_format  main_ext  '$remote_addr - $remote_user [$time_local] "$request" '
                          '$status $body_bytes_sent "$http_referer" '
                          '"$http_user_agent" "$http_x_forwarded_for" '
                          '"$host" sn="$server_name" '
                          'rt=$request_time '
                          'ua="$upstream_addr" us="$upstream_status" '
                          'ut="$upstream_response_time" ul="$upstream_response_length" '
                          'cs=$upstream_cache_status' ;

    access_log  /var/log/nginx/access.log  main_ext;
    error_log  /var/log/nginx/error.log warn;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    gzip  on;

    include /etc/nginx/conf.d/*.conf;
}

site is defined in the following way:

server {
        listen 80;
        listen [::]:80;

        server_name some.org www.some.org;

        location / {
                rewrite ^ https://$host$request_uri? permanent;
        }
}

server {
        listen 443 ssl http2;
        listen [::]:443 ssl http2;
        server_name some.org www.some.org;

        index index.php index.html index.htm;

        root /var/www/html;

        server_tokens off;

        ssl_certificate /etc/letsencrypt/live/some.org/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/some.org/privkey.pem;

        add_header X-Frame-Options "SAMEORIGIN" always;
        add_header X-XSS-Protection "1; mode=block" always;
        add_header X-Content-Type-Options "nosniff" always;
        add_header Referrer-Policy "no-referrer-when-downgrade" always;
        add_header Content-Security-Policy "default-src * data: 'unsafe-eval' 'unsafe-inline'" always;
        # add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload" always;
        # enable strict transport security only if you understand the implications

        location / {
                proxy_connect_timeout 600;
                proxy_send_timeout    600;
                proxy_read_timeout    600;
                proxy_redirect        off;
                proxy_pass http://wordpress;

                proxy_set_header      X-Real-IP $remote_addr;
                proxy_set_header      X-Forwarded-Proto https;
                proxy_set_header      X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header      Host $host;
        }

        location = /favicon.ico {
                log_not_found off; access_log off;
        }

        location = /robots.txt {
                log_not_found off; access_log off; allow all;
        }

        location ~* \.(css|js|gif|ico|jpeg|jpg|png)$ {
                expires max;
                log_not_found off;
        }
}

The whole stack works as expected, unless 10-15 users coming to website and trying to perform actions.. In this case some requests starts to hang (it is usually same POST request to some component) and after 3-4 minutes it is being released without any error and user can actually see the result of it. During the hanging site becomes not responsible from the same browser (but from others everything is ok!). Once request released - site is responsible again.

Logs are also pretty weird:

  • Once hanging/pending request comes to server it's being displayed in nginx access logs but not in wordpress logs
  • Once request is finally released (i.e. handled) it is displayed for second time in nginx access logs and in wordpress logs but times are different

nginx access logs:

62.96.39.243 - - [17/Jan/2020:10:56:41 +0000] "GET /something

78.43.40.52 - - [17/Jan/2020:10:56:41 +0000] "POST /ajax-bidsform.html?meth=post&yid=d7f9f1a1a0bf HTTP/2.0" 200 208 "https://some.org/xchange_XRP_to_SBERRUB/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.117 Safari/537.36" "-"

78.43.40.52 - - [17/Jan/2020:10:56:41 +0000] "GET /something

wordpress:

134.19.130.91 - - [17/Jan/2020:10:56:40 +0000] "GET /something

78.43.40.52 - - [17/Jan/2020:10:50:07 +0000] "POST /ajax-bidsform.html?meth=post&yid=d7f9f1a1a0bf HTTP/1.0" 200 559 "https://some.org/xchange_XRP_to_SBERRUB/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.117 Safari/537.36"

62.96.39.243 - - [17/Jan/2020:10:56:41 +0000] "GET /something

As you can see hanging POST /ajax-bidsform.html... has time of 10:56 in nginx logs, but 10:50 in wordpress - and this is exactly the time this request was done by client. As far as I understand this means that request was stuck somewhere on nginx level for almost 6 minutes until it actually was passed to wordpress. As you can see I have no nginx's ddos protection directives.

Also some notes from my side: during the hanging requests there are no any CPU or RAM long-term spikes, so it's likely not related to hardware issues. I was also thinking that it's somehow related to hanging script (i.e. ajax-bidsform.html) but it is started to happen only when we migrated to digital-ocean's cloud instance from virtual hosting (where it never happened before) so I guess it's a configurational issue. Timeline of requests in logs are also kinda proofs it.

So far I did try to:

  1. Increase worker_connections to 10000
  2. Increase nginx instance's (not host's one) net.core.somaxconn to 1024

But the issue is still happening.. Any ideas or thoughts would be very appreciated!

1 Answer 1

2

10:56 in nginx logs, but 10:50 in wordpress

I would interpret this as wordpress receiving the request at 10:50 and returning the result back to nginx at 10:56. To know for sure, you could add upstream_response_time to log_format in nginx. See Using NGINX Logging for Application Performance Monitoring.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .