Firstly, I know this is a loaded question, but I hoping that you guys will be able to point me in the right direction....
We've got a number of nginx-fronted web heads which site behind an AWS ELB. I've been running a number of load tests against our system and the page response is consistently between 140 & 180ms up until ~600 concurrent users, at which point it jumps to ~300ms and then increases inline with the number of concurrents going up. As the users are anonymous, all responses will be coming from the Nginx cache and not hitting the application (this has been confirmed by checking application requests logs on the web heads which aren't displaying any content).
I've played about with a variety of instance types and it seems to make very little difference. I have also changed various nginx & kernel (via sysctl) settings and it doesn't appear to make much difference either.
The reason for running the tests is we're trying an NFS shared cache for all Nginx heads, but the stats above are the same for both NFS share and individual, "on server" file caches.
Where am I going wrong?
Each Web Head consists of:
Nginx | File Cache | uWSGI | Django
The Testing Scenario is:
Duration: 1 minute Users: 1 - 2000 Urls: 1 per request which is a JSON API response.
The number of users increases to go from 1 concurrent -> 2000 concurrent over the course of a minute.
So basically, I would like to know how, and with what tools I can determine where the bottlenecks are? I know there are a number of different points where there might be limits/restrictions e.g. tcp buffer sizes, max connections and read/write for the cache files.
As requested, nginx configuration is as follows:
nginx.conf
user www-data;
worker_processes 4;
worker_rlimit_nofile 200000;
pid /run/nginx.pid;
events {
worker_connections 4000;
multi_accept on;
use epoll;
}
http {
##
# Basic Settings
##
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 30;
keepalive_requests 100000;
reset_timedout_connection on;
send_timeout 2;
types_hash_max_size 2048;
server_tokens off;
include /etc/nginx/mime.types;
default_type application/octet-stream;
file_cache max=200000 inactive=20s;
open_file_cache_valid 30s;
open_file_cache_min_uses 2;
open_file_cache_errors on;
##
# Logging Settings
##
access_log off;
error_log /var/log/nginx/error.log;
##
# Gzip Settings
##
gzip on;
gzip_disable "msie6";
gzip_proxied any;
gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;
include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}
site conf
uwsgi_cache_path /var/www/cache levels=1:2 keys_zone=STATIC:8m max_size=1000m inactive=600m;
uwsgi_temp_path /var/www/cache/tmp;
upstream server_ups {
server unix:/tmp/uwsgi.sock fail_timeout=0;
}
server {
listen 80;
client_max_body_size 4G;
uwsgi_buffers 8 16k;
uwsgi_buffer_size 32k;
uwsgi_read_timeout 300;
location / {
uwsgi_pass server_ups;
include uwsgi_params;
uwsgi_param Host $http_host;
uwsgi_param X-Real-IP $remote_addr;
uwsgi_param X-Forwarded-For $proxy_add_x_forwarded_for;
set $nocache 0;
if ($http_cookie ~ _ssa) {
set $nocache 1;
}
error_page 404 /404/;
error_page 500 502 503 504 /500/;
uwsgi_no_cache $nocache;
uwsgi_cache_bypass $nocache;
uwsgi_ignore_headers Set-Cookie;
uwsgi_ignore_headers Cache-Control;
uwsgi_ignore_headers X-Accel-Expires;
uwsgi_cache STATIC;
uwsgi_param X-Cache-Status $upstream_cache_status;
uwsgi_cache_key $http_host$request_uri;
uwsgi_cache_valid 200 30m;
uwsgi_cache_use_stale error timeout invalid_header updating http_500 http_503;
}
}
