Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We recently had some 500's from nginx itself that somehow were not logged (we have screenshots, but nothing in the logs). That is weird in itself, because usually errors show up there. Regardless, I am wondering if there is something like a connection pool size that if maxed out would result in a 500? We have correlated it potentially to a recent spike in traffic, but it is not conclusive.

Anyone have any ideas of how to begin to approach such an issue?

share|improve this question
    
The first two things you have to do are reproduce this error and find out the reason why the Nginx doesn't log to error_log. Also post your configuration file. –  quanta Oct 1 '11 at 11:25

2 Answers 2

up vote 3 down vote accepted

We use a combination of log formats in nginx and lmon to catch things like this. An NGINX log format like:

log_format main '$status:$request_time:$upstream_response_time:$pipe:$body_bytes_sent $connection $remote_addr $host $remote_user [$time_local] "$request" "$http_referer" "$http_user_agent" "$http_x_forwarded_for" $upstream_addr $upstream_cache_status "in: $http_cookie"'

Will capture a lot of helpful diagnostic info, like the upstream server that handled the request, as well as putting the status in the front so it is easy to read even if the logs are scrolling by pretty fast.

We use LMON to watch these logs and then alert us (pagers/email) if it sees errors, like 500s, 503s, 400s, in the logs:

http://www.bsdconsulting.no/tools/lmon-README

This can help you be alerted to an issue when its happening which is the easiest time to debug it.

The other thing you should probably consider if you haven't already is that by default nginx considers a 500 to be a fatal condition and doesn't try another upstream. If you have multiple upstreams you can configure it to use another one if it gets a 500, hopefully obscuring the failure from the user:

http://wiki.nginx.org/NginxHttpProxyModule#proxy_next_upstream

share|improve this answer
    
This is a very helpful answer, thank you! Off to implemenet proxy_next_upstream... –  kaleidomedallion Oct 1 '11 at 21:59

error_log $filename debug; will turn on debug level logging into the error log -- this will give you lots and lots of details of nginx's internal status at the time of the error, and if compiled with --with-debug (which several distros do by default) it'll give even more.

Be warned that the "debug" level really does generate lots of output, to the point that you may want to watch your disk space...

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.