Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We're using nginx as a reverse proxy to get files from upstream servers. The files are not dynamic – at least not on a per request basis – and are (sometimes) behind high latency.

I would like to limit the request for the same URL to exactly 1.

An example of the behaviour I want to avoid:

127.0.0.1 - - [03/Jan/2013:16:08:15 +0100] "GET /part-00132.ts HTTP/1.0" 200 1506068 "-" "Prefetch" "-"
127.0.0.1 - - [03/Jan/2013:16:08:28 +0100] "GET /part-00133.ts HTTP/1.0" 200 1189476 "-" "Prefetch" "-"
127.0.0.1 - - [03/Jan/2013:16:08:29 +0100] "GET /part-00133.ts HTTP/1.0" 200 1189476 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/536.26.17 (KHTML, like Gecko) Version/6.0.2 Safari/536.26.17" "-"
127.0.0.1 - - [03/Jan/2013:16:08:34 +0100] "GET /part-00133.ts HTTP/1.0" 200 1189476 "-" "Prefetch" "-"
127.0.0.1 - - [03/Jan/2013:16:08:44 +0100] "GET /part-00134.ts HTTP/1.0" 200 1762876 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/536.26.17 (KHTML, like Gecko) Version/6.0.2 Safari/536.26.17" "-"
127.0.0.1 - - [03/Jan/2013:16:08:53 +0100] "GET /part-00135.ts HTTP/1.0" 200 1627704 "-" "Prefetch" "-"
127.0.0.1 - - [03/Jan/2013:16:08:59 +0100] "GET /part-00136.ts HTTP/1.0" 200 1252456 "-" "Prefetch" "-"
127.0.0.1 - - [03/Jan/2013:16:09:03 +0100] "GET /part-00134.ts HTTP/1.0" 200 1762876 "-" "Prefetch" "-"
127.0.0.1 - - [03/Jan/2013:16:09:04 +0100] "GET /part-00137.ts HTTP/1.0" 200 1120292 "-" "Prefetch" "-"
127.0.0.1 - - [03/Jan/2013:16:09:07 +0100] "GET /part-00135.ts HTTP/1.0" 200 1627704 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/536.26.17 (KHTML, like Gecko) Version/6.0.2 Safari/536.26.17" "-"
127.0.0.1 - - [03/Jan/2013:16:09:12 +0100] "GET /part-00134.ts HTTP/1.0" 200 1762876 "-" "Prefetch" "-"
127.0.0.1 - - [03/Jan/2013:16:09:14 +0100] "GET /part-00136.ts HTTP/1.0" 200 1252456 "-" "Prefetch" "-"
127.0.0.1 - - [03/Jan/2013:16:09:17 +0100] "GET /part-00135.ts HTTP/1.0" 200 1627704 "-" "Prefetch" "-"
127.0.0.1 - - [03/Jan/2013:16:09:17 +0100] "GET /part-00138.ts HTTP/1.0" 200 1248884 "-" "Prefetch" "-"

It should be like this:

127.0.0.1 - - [03/Jan/2013:16:08:15 +0100] "GET /part-00132.ts HTTP/1.0" 200 1506068 "-" "Prefetch" "-"
127.0.0.1 - - [03/Jan/2013:16:08:28 +0100] "GET /part-00133.ts HTTP/1.0" 200 1189476 "-" "Prefetch" "-"
127.0.0.1 - - [03/Jan/2013:16:08:44 +0100] "GET /part-00134.ts HTTP/1.0" 200 1762876 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/536.26.17 (KHTML, like Gecko) Version/6.0.2 Safari/536.26.17" "-"
127.0.0.1 - - [03/Jan/2013:16:08:53 +0100] "GET /part-00135.ts HTTP/1.0" 200 1627704 "-" "Prefetch" "-"
127.0.0.1 - - [03/Jan/2013:16:08:59 +0100] "GET /part-00136.ts HTTP/1.0" 200 1252456 "-" "Prefetch" "-"
127.0.0.1 - - [03/Jan/2013:16:09:04 +0100] "GET /part-00137.ts HTTP/1.0" 200 1120292 "-" "Prefetch" "-"
127.0.0.1 - - [03/Jan/2013:16:09:17 +0100] "GET /part-00138.ts HTTP/1.0" 200 1248884 "-" "Prefetch" "-"

Update:

A diagram to describe the problem a little bit:

            nginx              upstream
           +-----+            +--------+
           |     |            |        |
   A +----->     +-----------------+   |
           |     |            |    |   |
   B +----->     +--------------+  |   |
           |     |            | |  |   |
     <-----+     <--------------|--+   |
           |     |            | |      |
     <-----+     <--------------+      |
           |     |            |        |
   C +----->     +---+        |        |
           |     |   |        |        |
     <-----+     <---+        |        |
           |     |            |        |
           +-----+            +--------+

So I want client B to wait for the A response.

share|improve this question
add comment

2 Answers

Taken from answer to Nginx how to disable multithread downloads:

Use the nginx example for the limit_conn configuration:

limit_zone one $binary_remote_addr 10m;

server {
  location /download/ {
    limit_conn one 1;
  }
}

It's not just download managers with Range requests that multiplex downloads (and this does not disable Range headers); modern browsers loading pages multiplex requests as well. Be very careful about using this configuration anywhere expect a location that you need to specifically prevent multiplexed downloads for.

share|improve this answer
    
Does this solve the problem of having client B's request forwarded to the backend before client A's response? –  Gert Jan 5 '13 at 14:58
add comment

One potential solution for you is caching. When using either proxy pass or fcgi pass with nginx, you have the option to cache the return data for a period of time. You can read more about it in the docs: proxy pass and fcgi version. The two are virtually identical to each other in how they work.

I also suggest looking up some tutorials on it since how the parts of the caching operatives work together aren't entirely trivial from the docs.

But the main idea is that you make a proxy "key" from your URL (in your case). And you can then set it to cache it for 10min or something. So the subsequent calls are served from the local cache instead of fetching it again.

You can then prevent subsequent requests during update by setting:

proxy_cache_use_stale updating;
share|improve this answer
    
This is already configured and happening. Subsequent requests (after the first successful one) are served from cache. The problem is the multiple requests until the first successful response. –  Gert Jan 3 '13 at 17:29
    
@Gert then why are you keep making unsuccessful responses? Or are you saying it requests again before the server even has the time to respond? –  Grumpy Jan 4 '13 at 18:57
    
a second request was made before the first response. I've added a diagram to make it clearer. –  Gert Jan 5 '13 at 14:57
    
@Gert, did you also use proxy_cache_use_stale updating option in nginx? –  Grumpy Jan 5 '13 at 19:17
    
the requests are for files for dynamically generated files. So we don't have a stale cache either. –  Gert Jan 6 '13 at 19:21
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.