9

There is problem with nginx. It closes connection before client finishes download. It looks like:

 $ wget -O /dev/null http://www.site.com/images/theme/front/clean.jpg
--2012-07-11 21:37:03--  http://www.site.com/images/theme/front/clean.jpg
Resolving www.site.com (www.site.com)... 123.234.123.234
Connecting to www.site.com (www.site.com)|123.234.123.234|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 90707 (89K) [image/jpeg]
Saving to: `/dev/null'

26% [===============>                    ] 24,291      --.-K/s   in 8.7s    

2012-07-11 21:37:12 (2.74 KB/s) - Connection closed at byte 24291. Retrying.

--2012-07-11 21:37:13--  (try: 2)  http://www.site.com/images/theme/front/clean.jpg
Connecting to www.site.com (www.site.com)|123.234.123.234|:80... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 90707 (89K), 66416 (65K) remaining [image/jpeg]
Saving to: `/dev/null'

53% [+++++++++++++++============>        ] 48,555      --.-K/s   in 8.7s    

2012-07-11 21:37:23 (2.74 KB/s) - Connection closed at byte 48555. Retrying.

--2012-07-11 21:37:25--  (try: 3)  http://www.site.com/images/theme/front/clean.jpg
Connecting to www.site.com (www.site.com)|123.234.123.234|:80... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 90707 (89K), 42152 (41K) remaining [image/jpeg]
Saving to: `/dev/null'

100%[+++++++++++++++++++++++++++========>] 90,707      --.-K/s   in 0.1s    

2012-07-11 21:37:25 (311 KB/s) - `/dev/null' saved [90707/90707]

at the same time with smaller images all is ok:

 $ wget -O /dev/null http://www.site.com/images/theme/front/grease.jpg
--2012-07-11 21:41:28--  http://www.site.com/images/theme/front/grease.jpg
Resolving www.site.com (www.site.com)... 123.234.123.234
Connecting to www.site.com (www.site.com)|123.234.123.234|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 21404 (21K) [image/jpeg]
Saving to: `/dev/null'

100%[====================================>] 21,404      --.-K/s   in 0.07s   

2012-07-11 21:41:29 (316 KB/s) - `/dev/null' saved [21404/21404]

This is the reason why this picture can't bee drawn fully size in browser. I can see only head of it.

Nginx is configured as front-end and apache as back-end. Direct link to apache works well, so there is problem in nginx. Am I right?

nginx config:

user satellite;
worker_processes  1;

error_log  /var/log/nginx/error.log;
pid        /var/run/nginx.pid;

events {
    worker_connections  1024;
    multi_accept on;
}

http {
    include       /etc/nginx/mime.types;
    access_log  /var/log/nginx/access.log;

    sendfile        on;
    keepalive_timeout  0;
    tcp_nodelay        on;

    gzip  on;
    gzip_disable "MSIE [1-6]\.(?!.*SV1)";
    client_max_body_size 100m;

    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-enabled/*;
    server {
            listen 123.234.123.234:80;
            server_name site.com www.site.com;
            location ~* ^/(admin/|dump/|webmail/|myadmin/|webim/) {
                    proxy_pass http://123.234.123.234:8080;
                    proxy_redirect http://site.com:8080/ /;
                    proxy_set_header Host $host;
                    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                    proxy_set_header X-Real-IP $remote_addr;
            }
            location / {
                    proxy_pass http://123.234.123.234:8080;
                    proxy_redirect http://site.com:8080/ /;
                    proxy_set_header Host $host;
                    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                    proxy_set_header X-Real-IP $remote_addr;
                    proxy_cache ino;
                    proxy_cache_valid 12h;
                    proxy_hide_header "Set-Cookie";
                    proxy_ignore_headers "Cache-Control" "Expires";
            }
            location ~* ^.+\.(jpg|swf|flv|ico|txt|jpeg|gif|png|svg|js|css|mp3|ogg|mpe?g|avi|zip|gz|bz2?|rar)$ {
                    access_log /home/satellite/logs/site.com.nginx.access.log;
                    error_page 404 = @fallback;
                    if ( $host ~* ^((.*).site.com)$ ) {
                            set $proot /home/satellite/www/$1;
                            break;
                    }
                    if ( $host = "www.site.com" ) {
                            break;
                    }
                    if ( $host = "site.com" ) {
                            break;
                    }

                    root /home/satellite/www/site.com;
            }
            location @fallback {
                    proxy_pass http://123.234.123.234:8080;
                    proxy_set_header Host $host;
                    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                    proxy_set_header X-Real-IP $remote_addr;
            }
    }

where I should dig to fix this issue?

2
  • 1
    Did you try to switch off sendfile?
    – VBart
    Jul 12, 2012 at 1:41
  • Yes, Nothing has changed.
    – rush
    Jul 12, 2012 at 4:46

6 Answers 6

10

The main thing I forgot is to check /var/log/nginx/error.log.

2012/07/12 08:46:44 [crit] 24074#0: *3 open() 
"/var/lib/nginx/proxy/1/00/0000000001" failed (13: Permission denied) 
while reading upstream, client: 109.173.96.30, server: site.com, request: 
"GET /images/theme/front/clean.jpg HTTP/1.1", upstream: 
"http://123.234.123.234:8080/images/theme/front/clean.jpg", 
host: "www.site.com", referrer: "http://www.google.com"

So I fixed /var/lib/nginx/proxy/* directories permissions (sudo chown -R www-data:www-data /var/lib/nginx/proxy/*) and now everything works great. Thank everybody for help.

1
  • 1
    Thanks for this. Seems like obvious advice but I hadn't checked either. In my case the cause was this: [crit] 6#6: *2577 mkdir() "/var/cache/nginx/proxy_temp/8" failed (28: No space left on device) while reading upstream Jun 6, 2019 at 6:58
1

A possible reason for the closing of the connection is a slow connection and a short keepalive_timeout. The default value for the keepalive_timeout is 75s. If it is too short and the connection is slow, then it could be closed too early.

http {
    ..
    keepalive_timeout 75;
}

One reason why your image download could be slow is your application. If you are using a Ruby-on-Rails application with an Asset Pipeline in combination with Nginx, then the image download could be slow because you are using the wrong image urls (without fingerprint generated by the asset pipeline). The Rails helpers asset_path and image_tag generate the right urls form files with fingerprint which can be downloaded quickly.

1

Another very important thing to check is: Make sure you have disk space left!

For me it was like following:

[user@server]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        30G   29G     0 100% /

For me, nginx was unable to write to the disk eventually causing the same issue! Hope it helps someone!

3
  • Please explain why you think lack of disk space can cause a TCP connection to close unexpectedly. And how opening a new TCP connection will temporarily solve that problem.
    – kasperd
    Nov 22, 2014 at 11:26
  • 1
    Yes, sure! Here is a scenario: - Nginx is working as a Proxy server - It's receiving a file from apache - Nginx receives the first chunk of the file (response code 206) - It writes the chunk to the hard drive and sends request for the next chunk - It receives the next chunk and fails write to hard drive as no space left! - Nginx closes the connection with response code 206. Full content was not served to the client! Hope that clarifies!
    – Ahsan
    Nov 23, 2014 at 3:52
  • 1
    In Rush's case the first image chunk of size 24,291 was successfully received. Which means up to ~ 24,291 can be served by nginx directly from memory. That's why the next image with size 21,404 was successfully served as it did not required HDD write.
    – Ahsan
    Nov 23, 2014 at 3:58
0

Your download rate is incredibly low. (2.74 KB/s!). Do you get the same result when running wget on the same machine where Nginx is located? It could be that Nginx is reacting reasonably to a request over a very slow link.

Otherwise, I recommend exploring the various time directives in the Nginx docs. Search for every mention of "timeout" on the page and see if you find that's a good match. For example, you can see you are timing out in what appears to be 10 seconds intervals, so any timeout that's about 10 seconds should receive extra scrutiny.

For example, the lingering_timeout directive defaults to 10 seconds, so you might check that.

You should also be looking into why the connection with your client is apparently so slow.

Another idea: Try an alternate client, like curl, and see you get the same result as you do with wget. If curl works fine, you should suspect there's something odd about wget is making the request.

1
  • That's not client trouble, cause even firefox has the same issue. I tried it from different machine in defferent geolocations. The same behaviour. Furthermore, as you can mention the speed is pretty good on small picture. ps. On machine where nginx is located everything is OK. So I will try to dig into timeout directive. pps. That's not network trouble, cause direct apache link to the same image works perfect.
    – rush
    Jul 11, 2012 at 20:37
0

Check the lingering_time value in nginx.conf. This will be by default set to 30sec. So what this will do is that, nginx will wait maximum of 30 sec for the client to send data.

If you are doing an upload or POST of a file which might take more than 30secs to complete, then the nginx server will reset the connection to client by sending a RST packet to client if the time to upload exceeds 30 secs.

For the nginx to wait for more time to listen to client data, then set this value to a higher value.

For detailed info regarding lingering_time, see it here lingering_time

0

I also resolved the issue the same way. I accidentally commented user www-data; in nginx config. Weird thing is I was proxying to WsgiDAV where uploads worked fine but downloads failed

2

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .