Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a lot of free RAM but my IO is always 100 %util or very close. What ways can I reduce IO by using more RAM? My iotop shows nginx worker processes with the highest io rate. This is a file server serving files ranging from 1mb to 2gb. Here is my nginx.conf

#user  nobody;
worker_processes  32;
worker_rlimit_nofile 10240;
worker_rlimit_sigpending 32768;

error_log  logs/error.log  crit;

#pid        logs/nginx.pid;


events {
    worker_connections  51200;
}


http {
    include       mime.types;
    default_type  application/octet-stream;

    access_log  off;
    limit_conn_log_level info;
    log_format  xfs  '$arg_id|$arg_usr|$remote_addr|$body_bytes_sent|$status';

    sendfile       off;
    tcp_nopush     off;
    tcp_nodelay on;
    directio 4m;
    output_buffers 3 512k;
    reset_timedout_connection on;
    open_file_cache max=5000 inactive=20s;
    open_file_cache_valid    30s;
    open_file_cache_min_uses 2;
    open_file_cache_errors   on;
    client_body_buffer_size 32k;

    server_tokens off;
    autoindex off;

    keepalive_timeout  0;
    #keepalive_timeout  65;
share|improve this question
2  
One solution to this is caching requests in memory. Depending on your setup (the server blocks), you may be able to use proxy_cache or fastcgi_cache with the cache path pointing to a ramdisk (tmpfs). You could also roll your own cache using nginx's memcached module. Alternatively, you can add a caching layer (e.g. Varnish) in front of nginx that will cache a copy of pages (and/or files) in memory and serve requests from there. –  cyberx86 Nov 17 '11 at 21:06
    
@cyberx86 post that as an answer and get some free rep :-) –  voretaq7 Nov 17 '11 at 22:06

4 Answers 4

up vote 4 down vote accepted

Turn sendfile on. Leave output_buffers default. Turn directio off. Add more RAM.

There is nothing better than linux VFS cache.

share|improve this answer
    
That's it, man. –  poige Nov 18 '11 at 6:57

It appears that you want to serve some requests directly from memory - which essentially amounts to caching those requests. In its basic form, you will process a request, store a copy in your memory based cache, and serve subsequent requests for the same resource from memory (until such a time as you invalidate/expire the cached copy).

The implementation of this will vary depending on the nature of your site (i.e. if you are serving very large files, it won't be as practical - if your content is highly dynamic it won't offer as much benefit, etc) - but for an 'average' site - with most of the page's i/o coming from static resources (images especially), it will reduce disk i/o (and should speed things up too).

If you are proxying to a backend server, you can use nginx's proxy_cache directive to cache pages. However, the cache location is typically on disk (which may speed things up, but may not save on disk i/o) - you can get around this by create a tmpfs mapped to memory (i.e. a ramdisk) in which to store your cache. Likewise, if you are using FastCGI, you can use fastcgi_cache.

Another, similar option (e.g. if you are not using nginx as proxy and are not using FastCGI) is to store pages in MemCached. Nginx comes with a memcached module, which will allow you to setup your own caching mechanism.

Alternatively, you could simply add a caching proxy in-front of Nginx (such as Varnish) - that will receive requests, and either serve them directly or pass the request onto Nginx. Varnish can be configured to store its cache in memory (or a file - although, that won't help your i/o) and will serve cache hits with a minimum of disk i/o.

share|improve this answer
    
This is a file server serving files ranging from 1mb to 2gb –  Matthew Hui Nov 17 '11 at 23:31
    
Depending on how much RAM you have and how often certain files are accessed, you could still use the caching idea. Essentially, you will place an upper limit on the size of files that are cached (e.g. 10MB), and cache the files that are accessed most often - it will make a difference to I/O. Also, depending on how compressible the files are, may be able to store gzipped versions of them, and use the gzip_static module to directly serve the compressed file. Of course, moving the files to a CDN would also lower I/O (but may defeat the point of the server). –  cyberx86 Nov 17 '11 at 23:45
    
I -1'd it — too many words, too little on the heart of the issue. –  poige Nov 18 '11 at 6:55

Please don't accept this answer as it's just one observation. But you use direct IO for files over 4M and only allow nginx 1.5MB of memory buffer for files.

directio 4m;
output_buffers 3 512k;

If I remember correctly then direct IO basically bypasses the file system IO cache to read directly from the HDD. So for files over 4M you use no memory to buffer the file and for files under 4M you use only 1.5M memory.

share|improve this answer

worker_processes 32;

— still you're guessing why?

I have a lot of free RAM but

I also would like to hear reasoning on having sendfile off; and using directio — am I the only one (surely not¹) who knows that directio (by definition) prohibits FS caching?

__ ¹ — Martin F, at least remembering this as well: Reducing IO caused by nginx

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.