I have the following stack running on Ubuntu 18.08 and defined as docker-compose:
- instance of
mariadb:10.3.20
- custom wordpress instance based on
wordpress:5.3.0-php7.2
with installedioncube
on it - custom nginx instance based on
nginx:1.13
with installednginx-amplify-agent
on it
nginx's config:
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
events {
worker_connections 10000;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
log_format main_ext '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for" '
'"$host" sn="$server_name" '
'rt=$request_time '
'ua="$upstream_addr" us="$upstream_status" '
'ut="$upstream_response_time" ul="$upstream_response_length" '
'cs=$upstream_cache_status' ;
access_log /var/log/nginx/access.log main_ext;
error_log /var/log/nginx/error.log warn;
sendfile on;
#tcp_nopush on;
keepalive_timeout 65;
gzip on;
include /etc/nginx/conf.d/*.conf;
}
site is defined in the following way:
server {
listen 80;
listen [::]:80;
server_name some.org www.some.org;
location / {
rewrite ^ https://$host$request_uri? permanent;
}
}
server {
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name some.org www.some.org;
index index.php index.html index.htm;
root /var/www/html;
server_tokens off;
ssl_certificate /etc/letsencrypt/live/some.org/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/some.org/privkey.pem;
add_header X-Frame-Options "SAMEORIGIN" always;
add_header X-XSS-Protection "1; mode=block" always;
add_header X-Content-Type-Options "nosniff" always;
add_header Referrer-Policy "no-referrer-when-downgrade" always;
add_header Content-Security-Policy "default-src * data: 'unsafe-eval' 'unsafe-inline'" always;
# add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload" always;
# enable strict transport security only if you understand the implications
location / {
proxy_connect_timeout 600;
proxy_send_timeout 600;
proxy_read_timeout 600;
proxy_redirect off;
proxy_pass http://wordpress;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-Proto https;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host;
}
location = /favicon.ico {
log_not_found off; access_log off;
}
location = /robots.txt {
log_not_found off; access_log off; allow all;
}
location ~* \.(css|js|gif|ico|jpeg|jpg|png)$ {
expires max;
log_not_found off;
}
}
The whole stack works as expected, unless 10-15 users coming to website and trying to perform actions.. In this case some requests starts to hang (it is usually same POST request to some component) and after 3-4 minutes it is being released without any error and user can actually see the result of it. During the hanging site becomes not responsible from the same browser (but from others everything is ok!). Once request released - site is responsible again.
Logs are also pretty weird:
- Once hanging/pending request comes to server it's being displayed in nginx access logs but not in wordpress logs
- Once request is finally released (i.e. handled) it is displayed for second time in nginx access logs and in wordpress logs but times are different
nginx access logs:
62.96.39.243 - - [17/Jan/2020:10:56:41 +0000] "GET /something
78.43.40.52 - - [17/Jan/2020:10:56:41 +0000] "POST /ajax-bidsform.html?meth=post&yid=d7f9f1a1a0bf HTTP/2.0" 200 208 "https://some.org/xchange_XRP_to_SBERRUB/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.117 Safari/537.36" "-"
78.43.40.52 - - [17/Jan/2020:10:56:41 +0000] "GET /something
wordpress:
134.19.130.91 - - [17/Jan/2020:10:56:40 +0000] "GET /something
78.43.40.52 - - [17/Jan/2020:10:50:07 +0000] "POST /ajax-bidsform.html?meth=post&yid=d7f9f1a1a0bf HTTP/1.0" 200 559 "https://some.org/xchange_XRP_to_SBERRUB/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.117 Safari/537.36"
62.96.39.243 - - [17/Jan/2020:10:56:41 +0000] "GET /something
As you can see hanging POST /ajax-bidsform.html...
has time of 10:56 in nginx logs, but 10:50 in wordpress - and this is exactly the time this request was done by client. As far as I understand this means that request was stuck somewhere on nginx level for almost 6 minutes until it actually was passed to wordpress. As you can see I have no nginx's ddos protection directives.
Also some notes from my side: during the hanging requests there are no any CPU or RAM long-term spikes, so it's likely not related to hardware issues. I was also thinking that it's somehow related to hanging script (i.e. ajax-bidsform.html
) but it is started to happen only when we migrated to digital-ocean's cloud instance from virtual hosting (where it never happened before) so I guess it's a configurational issue. Timeline of requests in logs are also kinda proofs it.
So far I did try to:
- Increase
worker_connections
to 10000 - Increase nginx instance's (not host's one)
net.core.somaxconn
to 1024
But the issue is still happening.. Any ideas or thoughts would be very appreciated!