1

I have a lot of free RAM but my IO is always 100 %util or very close. What ways can I reduce IO by using more RAM? My iotop shows nginx worker processes with the highest io rate. This is a file server serving files ranging from 1mb to 2gb. Here is my nginx.conf

#user  nobody;
worker_processes  32;
worker_rlimit_nofile 10240;
worker_rlimit_sigpending 32768;

error_log  logs/error.log  crit;

#pid        logs/nginx.pid;


events {
    worker_connections  51200;
}


http {
    include       mime.types;
    default_type  application/octet-stream;

    access_log  off;
    limit_conn_log_level info;
    log_format  xfs  '$arg_id|$arg_usr|$remote_addr|$body_bytes_sent|$status';

    sendfile       off;
    tcp_nopush     off;
    tcp_nodelay on;
    directio 4m;
    output_buffers 3 512k;
    reset_timedout_connection on;
    open_file_cache max=5000 inactive=20s;
    open_file_cache_valid    30s;
    open_file_cache_min_uses 2;
    open_file_cache_errors   on;
    client_body_buffer_size 32k;

    server_tokens off;
    autoindex off;

    keepalive_timeout  0;
    #keepalive_timeout  65;
2
  • 2
    One solution to this is caching requests in memory. Depending on your setup (the server blocks), you may be able to use proxy_cache or fastcgi_cache with the cache path pointing to a ramdisk (tmpfs). You could also roll your own cache using nginx's memcached module. Alternatively, you can add a caching layer (e.g. Varnish) in front of nginx that will cache a copy of pages (and/or files) in memory and serve requests from there.
    – cyberx86
    Nov 17, 2011 at 21:06
  • @cyberx86 post that as an answer and get some free rep :-)
    – voretaq7
    Nov 17, 2011 at 22:06

5 Answers 5

6

Turn sendfile on. Leave output_buffers default. Turn directio off. Add more RAM.

There is nothing better than linux VFS cache.

1
  • That's it, man.
    – poige
    Nov 18, 2011 at 6:57
6

Please don't accept this answer as it's just one observation. But you use direct IO for files over 4M and only allow nginx 1.5MB of memory buffer for files.

directio 4m;
output_buffers 3 512k;

If I remember correctly then direct IO basically bypasses the file system IO cache to read directly from the HDD. So for files over 4M you use no memory to buffer the file and for files under 4M you use only 1.5M memory.

3

It appears that you want to serve some requests directly from memory - which essentially amounts to caching those requests. In its basic form, you will process a request, store a copy in your memory based cache, and serve subsequent requests for the same resource from memory (until such a time as you invalidate/expire the cached copy).

The implementation of this will vary depending on the nature of your site (i.e. if you are serving very large files, it won't be as practical - if your content is highly dynamic it won't offer as much benefit, etc) - but for an 'average' site - with most of the page's i/o coming from static resources (images especially), it will reduce disk i/o (and should speed things up too).

If you are proxying to a backend server, you can use nginx's proxy_cache directive to cache pages. However, the cache location is typically on disk (which may speed things up, but may not save on disk i/o) - you can get around this by create a tmpfs mapped to memory (i.e. a ramdisk) in which to store your cache. Likewise, if you are using FastCGI, you can use fastcgi_cache.

Another, similar option (e.g. if you are not using nginx as proxy and are not using FastCGI) is to store pages in MemCached. Nginx comes with a memcached module, which will allow you to setup your own caching mechanism.

Alternatively, you could simply add a caching proxy in-front of Nginx (such as Varnish) - that will receive requests, and either serve them directly or pass the request onto Nginx. Varnish can be configured to store its cache in memory (or a file - although, that won't help your i/o) and will serve cache hits with a minimum of disk i/o.

3
  • This is a file server serving files ranging from 1mb to 2gb Nov 17, 2011 at 23:31
  • Depending on how much RAM you have and how often certain files are accessed, you could still use the caching idea. Essentially, you will place an upper limit on the size of files that are cached (e.g. 10MB), and cache the files that are accessed most often - it will make a difference to I/O. Also, depending on how compressible the files are, may be able to store gzipped versions of them, and use the gzip_static module to directly serve the compressed file. Of course, moving the files to a CDN would also lower I/O (but may defeat the point of the server).
    – cyberx86
    Nov 17, 2011 at 23:45
  • I -1'd it — too many words, too little on the heart of the issue.
    – poige
    Nov 18, 2011 at 6:55
0

directio disables OS cache, yet I see this suggested by people when asked how to serve large files without higher i/o load.

The reason to use directio would be e.g. if wanting to avoid OS level cache for data already cached by the software itself e.g. mysql server. As double caching wastes resources.

Another reason would be if you dont have much system ram, and want to prioritise ram usage for other things, not the files been hosted, in which case directio is good to prevent those files been cached.

So if you want to minimise i/o for files been hosted via nginx, then do "not" use directio.

I dont have knowledge/experience of other parameters such as output_buffers and sendfile. So this is not a complete answer, but I hope this information helps.

-1

worker_processes 32;

— still you're guessing why?

I have a lot of free RAM but

I also would like to hear reasoning on having sendfile off; and using directio — am I the only one (surely not¹) who knows that directio (by definition) prohibits FS caching?

__ ¹ — Martin F, at least remembering this as well: Reducing IO caused by nginx

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .