Unxeplainably high disk IO caused by nginx worker processes

Question

I have just setup a ubuntu 12.04.2 LTS server that serves a big number of quite large static files. The configuration is the same as on another machine which works great. The other machine uses Ubuntu 11.10 with nginx 1.0.5 . The machine with the problem uses nginx 1.1.19 and it can hardly push around 20MB/s (but is on a 1Gbit dedicated line) with the iotop showing high disk IO by nginx. This is from iotop:

  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
 4569 be/4 www-data  754.61 K/s    0.00 B/s  0.00 % 99.99 % nginx: worker process
 4571 be/4 www-data 1257.69 K/s    0.00 B/s  0.00 % 99.99 % nginx: worker process
 4574 be/4 www-data    2.46 M/s    0.00 B/s  0.00 % 99.99 % nginx: worker process
 3951 be/4 www-data 1760.77 K/s    0.00 B/s  0.00 % 99.99 % nginx: worker process is     shutting down
 3950 be/4 www-data  503.08 K/s    0.00 B/s  0.00 % 99.99 % nginx: worker process is shutting down
 4573 be/4 www-data 2012.31 K/s    0.00 B/s  0.00 % 99.99 % nginx: worker process
 3952 be/4 www-data 1006.15 K/s    0.00 B/s  0.00 % 99.99 % nginx: worker process is shutting down
 3954 be/4 www-data 1760.77 K/s    0.00 B/s  0.00 % 99.99 % nginx: worker process is shutting down
 4572 be/4 www-data    4.05 M/s    0.00 B/s  0.00 % 99.99 % nginx: worker process
 3956 be/4 www-data    2.70 M/s    0.00 B/s  0.00 % 99.99 % nginx: worker process is shutting down
 3953 be/4 www-data  251.54 K/s    0.00 B/s  0.00 % 99.99 % nginx: worker process is shutting down
 4567 be/4 www-data    2.21 M/s    0.00 B/s  0.00 % 98.30 % nginx: worker process
 4570 be/4 www-data  754.61 K/s    0.00 B/s  0.00 % 97.91 % nginx: worker process
 3949 be/4 www-data 1006.15 K/s    0.00 B/s  0.00 % 88.21 % nginx: worker process is shutting down
 3955 be/4 www-data 1509.23 K/s    0.00 B/s  0.00 % 84.60 % nginx: worker process is shutting down

So for some reason those processes that try to shutdown cause the IO and the server goes in a almost non-responsive state, with the load growing as high as 5-6 (this is a dual core machine). The CPU utilisation meanwhile is aroun 0.5%

After restarting nginx everything is fine for some time and then this happens again.

This is the latest from the error log of nginx:

013/03/18 13:09:28 [alert] 3676#0: open socket #297 left in connection 145

and then this happens:

2013/03/18 13:10:11 [alert] 3749#0: 100 worker_connections are not enough

and this is the nginx.conf:

user www-data;
worker_processes 8;
worker_rlimit_nofile 20480;
pid /var/run/nginx.pid;

events {
    worker_connections 100;
    # multi_accept on;
}

http {

    ##
    # Basic Settings
    ##

    sendfile off;
        output_buffers 1 512k;
    tcp_nopush on;
    tcp_nodelay on;
    keepalive_timeout 5;
    types_hash_max_size 2048;

Any help will be highly appreciated!

EDIT:

Sendfile on and off makes no difference.

worker_rlimit_nofile == worker_connections makes no difference.

worker_processes changes nothing also.

smartctl shows no problems with the disk, however I tried with the second disk on this machine and still no difference.

Answer 1

Relatively recent HDDs can do 150MB/s (1.2Gbps) on sequential reads (and writes), but if you have several parallel reads / writes (even if each read itself is still sequential), then the throughput speed will easily drop 10×.

So, 20MB/s (160Mbps) sounds like a limitation of your HDD.

Perhaps the other server has an SSD, or has more memory, and has these files cached, but this one has something wrong on the cache side (probably little memory, but maybe wrongly optimised kernel settings).

In any case, this would likely sound like something that's outside of nginx control.

You might try increasing your nginx in-memory buffers by several folds in an attempt to make the reads a little more sequential, but if you only have one platter-based HDD (e.g. at most 150MB/s on a single sequential read, drop of several folds on multiple reads), and no caching gets utilised due to low memory, then you won't be able to push anywhere close to 1Gbps (128MB/s).

If you really need 1Gbps throughput: if most common files can be cached in memory, get more memory; else, get a fast SSD.

Answer 2

I would turn sendfile on:

 sendfile on

I think it avoids unnecessary copying and saves on system calls. Also, tcp_nopush will only be on if sendfile is on. See here.

Also, remove types_hash_max_size.