1

I have a Ubuntu 15.10 running nginx 1.9.3 for serving static content that are mostly images and video files. There are no other custom processes running that consume even 1% of serer's resources. There is no PHP processing and no SQL database connections happening whatsoever.

The output of netstat -an | wc -l is always in range of 700-1500 tcp connections. In 95% cases most users request to download a certain 3.5 MB video file (nginx access logs confirm this) that due to caching mechanisms of both nginx and linux itself I am sure is always stored in RAM. And indeed - output of iotop tells me that there is 0 - 0.5% disk read most of the time so I know that the problem I am getting to is not related to disk I/O.

Running nload for a period of 3 hours tells me that my traffic looks like this:

Curr: 103.41 MBit/s
Avg: 73.97 MBit/s
Min: 14.62 MBit/s
Max: 457.88 MBit/s
Ttl: 141357.91 GByte

Whenever the number of tcp connections goes higher than ~1200, users experience delays/timeouts when loading data from the server.

Let me expand on what I have observed about these 'delays/timeouts' so far:

I am almost certain that nginx configuration is not to blame since it's not just HTTP connections that get timeouts but even when simply pinging (ICMP) the server from another computer I get a lot of 'request timed out' results when the above mentioned tcp connection marker is reached.

When running Wireshark from Windows and loading this 3.5 MB video in browser (during mentioned high loads) I see that a lot of packets labeled with

[TCP ACKed unseen segment] [TCP Previous segment not captured]

that means some chunks (packets) of this video file are not delivered to me. This got me thinking that my problem is most likely related to either my server's NIC not being able to process all the outgoing traffic or there is a bandwidth/traffic limitation form the hosting company that hosts my server. However, after multiple inquiries they assured me that there is no traffic limit set from their side that would be a problem for my current bandwidth consumption.

The next test I did was capturing ping requests/replies originating from my Windows OS to my server on both sides - using Wireshark on windows and tcpdump on Ubuntu. The result is the following almost always (during high loads):

On Ubuntu I can see that every ICMP Ping Request packet is matched with a Ping Reply packet. Not a single problem there. On Windows, 5-6 out of 20 ICMP Ping Request packets did not receive their matching Ping Reply packets.

From this I know that the server on CPU level sees and handles the connections correctly and it's just after sending a packet (reply) to Network interface (NIC) or beyond there is a problem.

Here are my configurations for nginx, network interface and my server info:

nginx.conf

user www-data;
worker_processes auto;
pid /run/nginx.pid;

events {
    use epoll;
        worker_connections 20000;
    multi_accept on;
}

worker_rlimit_nofile 25000;

http {

    ##
    # Basic Settings
    ##

    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    keepalive_timeout 15;
    types_hash_max_size 2048;
    # server_tokens off;

    # server_names_hash_bucket_size 64;
    # server_name_in_redirect off;

    include /etc/nginx/mime.types;
    default_type application/octet-stream;

        client_max_body_size 500M; 

    ##
    # SSL Settings
    ##

    ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
    ssl_prefer_server_ciphers on;

    ##
    # Logging Settings
    ##

    access_log /var/log/nginx/access.log;
    #access_log off;
        error_log /var/log/nginx/error.log;

    ##
    # Gzip Settings
    ##

    gzip on;
    gzip_disable "msie6";

    # gzip_vary on;
    # gzip_proxied any;
    # gzip_comp_level 6;
    # gzip_buffers 16 8k;
    # gzip_http_version 1.1;
     gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

    ##
    # Virtual Host Configs
    ##


        open_file_cache          max=2000 inactive=20s;
        open_file_cache_valid    60s;
        open_file_cache_min_uses 5;
        open_file_cache_errors   off;        

    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-enabled/*;
}

/etc/nginx/sites-available/default

server {
    listen 80 backlog=4096 default_server;
    listen [::]:80 default_server;

    # SSL configuration
    #
    # listen 443 ssl default_server;
    # listen [::]:443 ssl default_server;
    #
    # Note: You should disable gzip for SSL traffic.
    # See: https://bugs.debian.org/773332
    #
    # Read up on ssl_ciphers to ensure a secure configuration.
    # See: https://bugs.debian.org/765782
    #
    # Self signed certs generated by the ssl-cert package
    # Don't use them in a production server!
    #
    # include snippets/snakeoil.conf;

    root /var/www/html;

    # Add index.php to the list if you are using PHP
    index index.php index.html index.htm index.nginx-debian.html;

    server_name 93.188.8.60;

    location / {
        # First attempt to serve request as file, then
        # as directory, then fall back to displaying a 404.
        try_files $uri $uri/ =404;
    }

    # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
    #
    location ~ \.php$ {
        include snippets/fastcgi-php.conf;
    #
    #   # With php5-cgi alone:
    #   fastcgi_pass 127.0.0.1:9000;
    #   # With php5-fpm:
        fastcgi_read_timeout 900; 
                fastcgi_pass unix:/var/run/php5-fpm.sock;
    }

    # deny access to .htaccess files, if Apache's document root
    # concurs with nginx's one
    #
    location ~ /\.ht {
        deny all;
    }
}

Output of ifconfig (note: I have modified txqueuelen parameter when trying various NIC optimizations to fix this problem)

eno1      Link encap:Ethernet  HWaddr 30:5a:3a:56:39:bc
          inet addr:93.188.8.60  Bcast:93.188.8.255  Mask:255.255.255.0
          inet6 addr: fe80::325a:3aff:fe56:39bc/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:50327240672 errors:0 dropped:173244 overruns:0 frame:0
          TX packets:104203043819 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:5000
          RX bytes:4067795598631 (4.0 TB)  TX bytes:151795336718586 (151.7 TB)
          Interrupt:20 Memory:fb200000-fb220000

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:133764933 errors:0 dropped:0 overruns:0 frame:0
          TX packets:133764933 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:10569844193 (10.5 GB)  TX bytes:10569844193 (10.5 GB)

Output of sysctl -a

https://pastebin.com/zv86U9jn

Note: I have modified several parameters such as:

net.core.somaxconn
net.core.netdev_max_backlog
net.ipv4.tcp_max_syn_backlog 

In attempts to combat this issue.

I have also set ulimit -n 900000 back when I though this issue could have been related to the number of open files each process is allowed to have.

My Server specs:

Motherboard: ASUS X99-A

CPU: I am having trouble determining exact Core i7 model for CPU due to the fact that 'model name' from cat /proc/cpuinfo outputs Genuine Intel(R) CPU @ 2.40GHz but I know it has 24 cores and here is the output from cat /proc/cpuinfo: https://pastebin.com/TR7aS1NM

RAM: 64 GB

HDD: 4TB Seagate

NIC: Onboard Intel® I218V, 1 x Gigabit LAN Controller

I have been struggling with this problem for 1 week now. Is NIC the cause of this issue? Perhaps I am mistaken in my assumptions? I can provide any other command output required.

  • You could use iperf tool to measure how much throughput you can get between your server and some client. This way you can eliminate any other causes for issues. – Tero Kilkanen Sep 16 '17 at 15:04
  • iperf 's documentation looks rather promising. Thank you I will try that – astralmaster Sep 16 '17 at 16:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.