Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have an nginx reverse-proxy which proxies requests from an outer amazon ELB to internal ELBs.

I have 6 backend instances that handles the requests. The site-enabled configs looks like this, but there are different port numbers and proxy_pass. Everything else is identical:

server {
    listen 3000;
    location / {
            proxy_pass http://internal-prod732r8-PrivateE-1GJ070M0745TT-348518554.eu-west-1.elb.amazonaws.com:3000;
            include /etc/nginx/proxy.conf;
    }

}

Once about every 24h one of the configurations stops working. All other proxies works fine. If i restart nginx all configurations works again. There is nothing in error.log, nothing weird in access log, syslog or dmesg.

Is this something known? Have i done something wrong with my proxy configs? Are there any other logs i can look in?

share|improve this question
add comment

2 Answers

If your proxy_pass did not pass directly to one URL like your example shows (http://amazonaws.com), but instead to a proxy upstream farm, like this:

upstream my_upstream {
 server1 127.0.0.1:1337;
 server2 127.0.0.1:1338; 
}
location / {
 proxy_pass         http://my_upstream;
}

Then you will less concerned about one of the upstreams temporarily failing. Because they will all be doing the same job. If one fails to reply, then the next one will be proxied for that response. Peace of mind.

Nginx will skip a failed machine for x seconds automaticaly. Until you repair it, or until it returns by itself. (http://wiki.nginx.org/HttpUpstreamModule)

So whatever the reason for your interruptions may be, by distributing them on an upstream farm, this converts into an easier setup.

share|improve this answer
    
Thanks for your reply! The weird thing is that i can do a request to the backend instance directly but not through nginx. If i just restart nginx the request is proxied again. Since this is already in production environment i really want to find out why one of the configs seems to be "unloaded" or how i can find out what nginx really does behind the scenes. –  user202172 Dec 11 '13 at 12:04
    
You might want to hunt for more nginx logging info then. This is an issue where someone tried, like you, to find out more about "intermittent issues [...] I am proxying" stackoverflow.com/questions/9914792/… He describes a way of pulling more relevant logs. Hope it helps. –  user18099 Dec 11 '13 at 12:13
add comment
up vote 0 down vote accepted

The answer to this question is that ELBs sometimes change ip adresses and nginx does name resolving during start.

To fix this there is always a DNS server in your VPC at 0.2. So if the local ip CIDR is 10.0.0.0/16 the DNS server is at 10.0.0.2.

Add this to the nginx config.

resolver 10.0.0.2 valid=10s;

The proxy_pass also need to contain a variable so based on the configuration above this is the correct config:

server {
    listen 3000;
    location / {
            resolver 10.0.0.2 valid=10s;
            set $backend "http://internal-prod732r8-PrivateE-1GJ070M0745TT-348518554.eu-west-1.elb.amazonaws.com:3000"
            proxy_pass $backend;
            include /etc/nginx/proxy.conf;
    }
}
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.