0

Firstly, I know this is a loaded question, but I hoping that you guys will be able to point me in the right direction....

We've got a number of nginx-fronted web heads which site behind an AWS ELB. I've been running a number of load tests against our system and the page response is consistently between 140 & 180ms up until ~600 concurrent users, at which point it jumps to ~300ms and then increases inline with the number of concurrents going up. As the users are anonymous, all responses will be coming from the Nginx cache and not hitting the application (this has been confirmed by checking application requests logs on the web heads which aren't displaying any content).

I've played about with a variety of instance types and it seems to make very little difference. I have also changed various nginx & kernel (via sysctl) settings and it doesn't appear to make much difference either.

The reason for running the tests is we're trying an NFS shared cache for all Nginx heads, but the stats above are the same for both NFS share and individual, "on server" file caches.

Where am I going wrong?

Each Web Head consists of:

Nginx | File Cache | uWSGI | Django

The Testing Scenario is:

Duration: 1 minute Users: 1 - 2000 Urls: 1 per request which is a JSON API response.

The number of users increases to go from 1 concurrent -> 2000 concurrent over the course of a minute.

So basically, I would like to know how, and with what tools I can determine where the bottlenecks are? I know there are a number of different points where there might be limits/restrictions e.g. tcp buffer sizes, max connections and read/write for the cache files.

As requested, nginx configuration is as follows:

nginx.conf

user www-data;
worker_processes 4;
worker_rlimit_nofile 200000;
pid /run/nginx.pid;

events {
    worker_connections 4000;
    multi_accept on;
    use epoll;
}

http {

    ##
    # Basic Settings
    ##

    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    keepalive_timeout 30;
    keepalive_requests 100000;
    reset_timedout_connection on;
    send_timeout 2;
    types_hash_max_size 2048;
    server_tokens off;

    include /etc/nginx/mime.types;
    default_type application/octet-stream;

    file_cache max=200000 inactive=20s; 
    open_file_cache_valid 30s; 
    open_file_cache_min_uses 2;
    open_file_cache_errors on;

    ##
    # Logging Settings
    ##

    access_log off;
    error_log /var/log/nginx/error.log;

    ##
    # Gzip Settings
    ##

    gzip on;
    gzip_disable "msie6";
    gzip_proxied any;
    gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;


    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-enabled/*;
}

site conf

uwsgi_cache_path  /var/www/cache levels=1:2 keys_zone=STATIC:8m max_size=1000m inactive=600m;
uwsgi_temp_path /var/www/cache/tmp;

upstream server_ups { 
    server unix:/tmp/uwsgi.sock fail_timeout=0;
}

server {
    listen   80;
    client_max_body_size 4G;

    uwsgi_buffers 8 16k;
    uwsgi_buffer_size 32k;
    uwsgi_read_timeout 300;

    location / {
        uwsgi_pass server_ups;
        include uwsgi_params;
        uwsgi_param   Host             $http_host;
        uwsgi_param   X-Real-IP        $remote_addr;
        uwsgi_param   X-Forwarded-For  $proxy_add_x_forwarded_for;

        set $nocache 0;

        if ($http_cookie ~ _ssa) {
            set $nocache 1;
        }

        error_page 404 /404/;
        error_page 500 502 503 504 /500/;

        uwsgi_no_cache $nocache;
        uwsgi_cache_bypass $nocache;

        uwsgi_ignore_headers Set-Cookie;
        uwsgi_ignore_headers Cache-Control;
        uwsgi_ignore_headers X-Accel-Expires;
        uwsgi_cache STATIC;
        uwsgi_param X-Cache-Status $upstream_cache_status;
        uwsgi_cache_key $http_host$request_uri;
        uwsgi_cache_valid      200  30m;
        uwsgi_cache_use_stale  error timeout invalid_header updating http_500 http_503;
    }
}
  • This is far too broad to be answered. It's impossible to help you without knowing the load-test ramp-up and duration, user session duration, hits per user, resource types, etc. In case of performance issues when concurrency increases, the first thing that you should look at is your app, not nginx. – Xavier Lucas Feb 18 '15 at 15:50
  • I have added the load test ramp up, but I would argue, that as the application isn't getting hit, as the responses are coming from Nginx's cache, the app isn't the problem. – justcompile Feb 18 '15 at 15:55
  • How do you check that the cache is hit and not the app ? How many URIs per user ? How many of them should be answered from the cache and how many are standard static files requests ? Maximum hits per second ? What's your full nginx configuration ? How many nginx instances ? What are you storing in the NFS export and how is it mounted on web servers ? – Xavier Lucas Feb 18 '15 at 16:13
  • I've added some more information, but I've tried with 2 or 4 Nginx instances, no different. I'm currently running tests with local caches on each web server rather than via NFS as I wanted to test the bottleneck wasn't being caused by NFS – justcompile Feb 18 '15 at 16:29
0

Use performance monitoring tools. Try to collect metrics and graph them. Nothing beats graphs.

A tool like Munin can be very helpful in these situations. Look at memory, io, processes, cpu, networking, interrupts, nfs, etc. over time. Export nginx metrics too and graph them.

http://munin-monitoring.org/

Also bear in mind that when benchmarking from a single client machine you can be hitting client side bottlenecks or limits.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.