Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We recently moved to Nginx for serving static images for our website. The entry level server is easily able to handle the load with load avg mostly around 0 and low cpu usage.

But when we logged nginx $request_time in access logs, we're seeing some requests taking a long time even a few seconds, given that these are static images with no cgi, the long request time is odd. I looked at IO rates using iotop and they are pretty low too. The request time looks correlated to image sizes from a quick look at sorted data but there are some smaller images with longer request times.

What could be the reason for this?

I'm guessing this could be related to either disk seek time or TCP/IP tuning. How can it be debugged?

UPDATE: adding some config files

sysctl

net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_sack = 1
net.ipv4.tcp_retrans_collapse = 1
net.ipv4.ip_default_ttl = 64
net.ipv4.ip_no_pmtu_disc = 0
net.ipv4.ip_nonlocal_bind = 0
net.ipv4.tcp_syn_retries = 5
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_max_orphans = 262144
net.ipv4.tcp_max_tw_buckets = 262144
net.ipv4.ip_dynaddr = 0
net.ipv4.tcp_keepalive_time = 7200
net.ipv4.tcp_keepalive_probes = 9
net.ipv4.tcp_keepalive_intvl = 75
net.ipv4.tcp_retries1 = 3
net.ipv4.tcp_retries2 = 15
net.ipv4.tcp_fin_timeout = 60
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_tw_recycle = 0
net.ipv4.tcp_abort_on_overflow = 0
net.ipv4.tcp_stdurg = 0
net.ipv4.tcp_rfc1337 = 0
net.ipv4.tcp_max_syn_backlog = 2048
net.ipv4.ip_local_port_range = 32768    61000
net.ipv4.igmp_max_memberships = 20
net.ipv4.igmp_max_msf = 10
net.ipv4.inet_peer_threshold = 65664
net.ipv4.inet_peer_minttl = 120
net.ipv4.inet_peer_maxttl = 600
net.ipv4.inet_peer_gc_mintime = 10
net.ipv4.inet_peer_gc_maxtime = 120
net.ipv4.tcp_orphan_retries = 0
net.ipv4.tcp_fack = 1
net.ipv4.tcp_reordering = 3
net.ipv4.tcp_ecn = 2
net.ipv4.tcp_dsack = 1
net.ipv4.tcp_mem = 1540608  2054144 3081216
net.ipv4.tcp_wmem = 4096    65536   16777216
net.ipv4.tcp_rmem = 4096    87380   16777216
net.ipv4.tcp_app_win = 31
net.ipv4.tcp_adv_win_scale = 2
net.ipv4.tcp_tw_reuse = 0
net.ipv4.tcp_frto = 2
net.ipv4.tcp_frto_response = 0
net.ipv4.tcp_low_latency = 0
net.ipv4.tcp_no_metrics_save = 0
net.ipv4.tcp_moderate_rcvbuf = 1
net.ipv4.tcp_tso_win_divisor = 3
net.ipv4.tcp_congestion_control = cubic
net.ipv4.tcp_abc = 0
net.ipv4.tcp_mtu_probing = 0
net.ipv4.tcp_base_mss = 512
net.ipv4.tcp_workaround_signed_windows = 0
net.ipv4.tcp_dma_copybreak = 4096
net.ipv4.tcp_slow_start_after_idle = 1
net.ipv4.tcp_available_congestion_control = cubic reno
net.ipv4.tcp_allowed_congestion_control = cubic reno

iostat Linux 2.6.32-5-amd64 (sg1) 04/18/2013 _x86_64_ (8 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.66    0.00    0.74    0.56    0.00   98.05

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              14.82       273.38       621.38  368357866  837247732

nginx.conf

user www-data;
worker_processes  8;
worker_rlimit_nofile 200000;

events {
    worker_connections  10240;
    multi_accept on;
    use epoll;
}

http {
    include       /etc/nginx/mime.types;

    access_log  /var/log/nginx/access.log;

    sendfile        on;
    tcp_nopush      on;

    keepalive_timeout  15;
    tcp_nodelay        on;

    open_file_cache max=400000 inactive=30s;
    open_file_cache_valid 120s;
    open_file_cache_min_uses 2;
    open_file_cache_errors on;


    log_format timed_combined '$remote_addr - $remote_user [$time_local]  '
    '"$request" $status $body_bytes_sent '
    '"$http_referer" "$http_user_agent" '
    '$request_time $upstream_response_time $pipe';


    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-enabled/*;

}
share|improve this question
1  
Without looking at logs it's really a guessing game. Also sever specs would help. Is this a shared server? More information please, anything you can think of. –  Drew Khoury Apr 18 '13 at 6:38
    
As above, logs please. Also take a look at iostat to get IO usage/latency. Other useful stuff; what sort of disks are you using (local disks in RAID, SAN, SSDs, disk speed etc.), how big are the files? Is it happening to the same images all the time? maybe post a bit of your NGINX config too, otherwise most of our suggestions will be guesswork –  sam Apr 18 '13 at 6:42
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.