Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm currently receiving a fairly large HTTP flood right now, and it's causing my nginx reverse proxy to produce a 502 Bad Gateway.

I have a frontend server running nginx as a proxy to my backend server, but it's just getting a bunch of connect() failed (110: Connection timed out) while connecting to upstream errors. Tons of them. If I bypass the proxy server to connect to the backend, I can run the site just fine, so I know it's in the reverse proxy somewhere. However, I have no idea how to determine why it's timing out.

Any help?

running nginx 1.2.3 on CentOS 6.2

share|improve this question
    
You could start by updating Nginx to the latest version. Albeit, I'm not aware of such a bug in 1.2.3 –  sonassi Oct 1 '12 at 1:01
1  
....and then have a look at whatever is REFUSING THE CONNECIONS FROM NGINX –  symcbean Oct 1 '12 at 12:33
    
What is your backend server? I've been confused before by errors when the error Nginx was serving up was actually coming from the backend. Does not seem like the case here, but you need to update your question with more details. –  jeffatrackaid Oct 2 '12 at 18:44
    
Also, are you connecting over a private/public network to the backend? Are the IPs of the proxy whitelisted in any firewalls, ddos, or other ip/rate-limiting type tools? What does a netstat on the backend server look like? How many connections are open? What is MaxClients on the backend? Are you exhausting those? –  jeffatrackaid Oct 2 '12 at 18:45
add comment

4 Answers

up vote 2 down vote accepted
+100

I'm assuming you've already jacked your Nginx error logging level up to debug. If not, start there.

Your best bet is probably going to be using strace to view the system calls being made by Nginx. In particular, you'll want to pay attention to connect() calls, and keep an eye on the return codes of these (man 2 connect can be your friend here).

Once you have that information, you can better make an educated guess about whether the issue is confined to your frontend proxy, or has something to do with the interactions between the proxy and backend application server.

share|improve this answer
add comment

It doesn't get much more pedantic than this unless you want to put in dtrace probes:

  1. Set debug log level: /etc/nginx/nginx.conf:

    ...
    http {
            ...
            error_log error_log /var/log/nginx/error.log debug; # todo testing remove me not for production use
            ...
    }
    
  2. Setup tcpdump in another window:

    tcpdump not port 22 -vvv -s0 -q -XXX
    
  3. Monitor log files in yet another window:

    tail -f /var/log/nginx/*
    
  4. Startup nginx interactively with strace:

    # top of /etc/nginx/nginx.conf:
    
    daemon off; # todo testing remove me not for production use
    

    And then

     $ strace nginx 
    

Further debugging can be had with an nginx compiled with --with-debug. Check it by running:

    nginx -V 2>&1 | grep -- '--with-debug' # no output if not debug

Another good module not compiled by default is: HttpStubStatusModule. In all likelihood, any decent setup will require a custom-compiled nginx (highly-recommend packaging using distro's packaging tools).

Most of these are unsuitable for production use, look at compiling nginx with gperf if you need more stats.

share|improve this answer
add comment

Looks like you are debugging a high traffic site.

Use debug with debug_connection directive so nginx error log will show debug logs from your IP only.

Once you start seeing some useful error logs rather than activating debug option for entire nginx config, add a separate error_log /path/to/some/file/ debug; directive in location {..} block responsible for reverse_proxy connection.

This way you will be able to isolate debug error log from your IP only.

Try to relate it with request you are making (from your browser).

For example, please check: http://rtcamp.com/tutorials/debugging-nginx-configuration/

A level ahead, you can use Nginx's HttpEchoModule

share|improve this answer
add comment

I've never found Nginx to be a bottleneck, in most cases its more than capable than the back ends. But if you tested without Nginx and found no error, then its going to be either (or both):

  1. Nginx configuration issue
    1. Upstream time out value wrong
    2. Wrong probe URL on upstream
    3. Too few workers
    4. Etc.
  2. Operating system TCP/IP bottleneck
    1. It could be that the proxying itself is causing a duplication of open ports and states. Be it file descriptors, ports, TCP connections

Without seeing your Nginx configs, no-one can comment on the former. And without suitable outputs from the OS, no-one can comment on the latter.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.