0

I have some file hosting servers running nginx, serving static files which size average from 500MB to 6GB. The servers use Lustre1.8 as cluster filesystem. Files stay on some raid6 array with stripe size = 512KB.

At normal situation, nginx works very well.

_ The disk i/o : Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn

sdc 727.00 136.23 5.75 136 5

sdd 627.00 124.36 0.01 124 0

_ Traffic out :

iface Rx Tx Total

eth2: 480.00 b/s 551.30 Mb/s 551.30 Mb/s

eth3: 480.00 b/s 481.72 Mb/s 481.72 Mb/s

eth4: 136.47 Mb/s 525.45 Mb/s 661.92 Mb/s

eth5: 480.00 b/s 497.82 Mb/s 497.82 Mb/s

bond0: 136.47 Mb/s 2.01 Gb/s 2.14 Gb/s

_ Number of files are serving :

lsof -u nginx -n | grep storagefile | wc -l

3982

But, when in hot situation (when a new hot file, may be a porn dvd or JAV recently uploaded), there are a lot of clients (more than 800) download that file at the same time, and nginx processes are going to D state. At last, nearly all processes are D state ! This makes the download speed very slow :(

root 25821 0.0 0.0 33032 584 ? Ss Dec20 0:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf

nginx 25823 0.8 1.5 158384 126396 ? S Dec20 23:53 nginx: worker process

nginx 25824 0.7 1.8 182276 150420 ? D Dec20 21:58 nginx: worker process

nginx 25825 0.7 2.1 207584 175728 ? D Dec20 22:01 nginx: worker process

nginx 25826 0.8 1.8 186052 154196 ? D Dec20 23:32 nginx: worker process

nginx 25827 0.7 1.9 191448 159464 ? D Dec20 23:03 nginx: worker process

nginx 25828 0.8 1.6 166044 134188 ? D Dec20 24:56 nginx: worker process

nginx 25829 0.7 1.3 139308 107452 ? S Dec20 23:00 nginx: worker process

nginx 25830 0.7 1.7 176652 144796 ? D Dec20 21:08 nginx: worker process

nginx 25832 0.7 1.2 136648 104788 ? D Dec20 20:25 nginx: worker process

nginx 25833 0.8 1.7 178948 146964 ? D Dec20 23:27 nginx: worker process

nginx 25834 0.7 2.0 195828 163968 ? D Dec20 21:45 nginx: worker process

nginx 25835 0.8 1.6 166200 134344 ? S Dec20 23:30 nginx: worker process

nginx 25836 0.8 1.3 144624 112640 ? D Dec20 23:50 nginx: worker process

nginx 25837 0.7 1.3 143644 111784 ? D Dec20 22:02 nginx: worker process

nginx 25838 0.7 1.3 141912 110056 ? D Dec20 21:17 nginx: worker process

nginx 25839 0.6 1.4 150580 118724 ? S Dec20 20:12 nginx: worker process

nginx 25840 0.8 1.5 158916 126928 ? D Dec20 23:48 nginx: worker process

...

I have tried many time tuning the number of process worker, but I didn't work ! How can I fix this ? I think when a lot off client access the same file, it suppose to be better because of caching ?!

Here is the config of nginx :

#

worker_processes 48; worker_rlimit_nofile 800000; events { worker_connections 51200; use epoll; } http { sendfile off; directio 1m; output_buffers 1 512k;

tcp_nopush off; tcp_nodelay on;

keepalive_timeout 5; ...

nginx -V

nginx: nginx version: nginx/1.0.0 nginx: built by gcc 4.1.2 20080704 (Red Hat 4.1.2-50)

uname -a

Linux OST 2.6.18-194.17.1.el5_lustre.1.8.5 #1 SMP Mon Nov 15 15:48:43 MST 20

If you need more info, tell me.

1 Answer 1

1

Your disk subsystem is unable to provide enough speed, that's why nginx is being locked on IO, because it's syncronous by default.

You may try to use aio: http://wiki.nginx.org/HttpCoreModule#aio - it would be solution to worker processes locking, so all other requests that are related to proxying or other asyncronus operations will work faster. But aio will not sightly speedup your downloads.

Consider adding more memory: increasing from 8 to 24 GB will improve linux vfs caching and it may give icredible results.

RAID6 is not good for static file serving. It shows good speed while single thread reading and performance is degraded on tens and hundreds concurent readings. There is nothing better than separate disks configured as JBOD. You may also try to use SSD for some "hot" content.

3
  • Thanks for replying. In hot time, when most of process turn into D state, I have tried cp/rsync/dd that "hot file" to /dev/null or to another ftpserver to measure that if there's bottle neck at disk i/o ?! But it was not disk i/o, cp/rsync/dd was done at very good speed (about 100~200Mbyte/s) => So I dont think the problem at disk i/o but nignx (directIO) not good at high concurent reading. I also try AIO but the traffic out lower than directIO very much (about 10% compares to DirectIO).
    – Atrus
    Dec 22, 2011 at 6:58
  • remsys.com/nginx-on-1gbps
    – Atrus
    Dec 22, 2011 at 8:27
  • Try with "sendfile on;". Remove directio and output_buffers. DirectIO is not good if you have popular files with several readers. It forces nginx to read from disk instead of memory (vfs cache).
    – Vadim
    Dec 22, 2011 at 10:45

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .