1

We're using Nginx + PHP + FastCGI for our web server and we just saw a bunch of timeout errors (see below) in the error log. We quickly restarted nginx and that fixed the problem - for now. However, we're trying to find the root-cause and fix the problem so it doesn't re-occur - ever.

Specifically,

  • can you explain the error to us or point us in the right direction?
  • looks like we're hitting two errors together: 18657 (connect failed) and 18656 (upstream timed out). When do they occur?
  • is there a way to see the network statistics on the server when the error occurred? We'd like to see how many php processes, inbound connections, etc. we had when the problem hit us.
  • how can we reproduce the problem and fix it for sure?

Here are the errors (we saw hundreds of them, posting a handful):

  • error.log:2011/08/04 19:31:44 [error] 18657#0: *464859 connect() failed (110: Connection timed out) while connecting to upstream, client: 166.205.138.117, server: api.website.com, request: "POST /v3/logAgentEvent HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "api.website.com"
  • error.log:2011/08/04 19:31:44 [error] 18656#0: *464476 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 98.140.86.90, server: api.website.com, request: "POST /v3/logAgentEvent HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "api.website.com"
0

1 Answer 1

0

can you explain the error to us or point us in the right direction?

Check the documentation and/or source code of the component that is causing the error.

looks like we're hitting two errors together: 18657 (connect failed) and 18656 (upstream timed out).

  • 18657 (connect failed): Looks like this error is triggered by the FASTCGI implementation you're using. Check their documentation / source-code for the meaning.
  • 18656 (upstream timed out): Looks like this error is triggered by the FASTCGI implementation you're using. Check their documentation / source-code for the meaning.

When do they occur?

You find the timestamps in the logs when (in time) they occur. To understand why (when in meaning?) they occur, check the documentation / sourcecode. If that is too complicated for you, get in contact with your sysadmin and let it analyze for you. This can be very specific for the exact server version you're using, the exact FASTCGI implementation you're using and the exact PHP configuration you're using.

is there a way to see the network statistics on the server when the error occurred? We'd like to see how many php processes, inbound connections, etc. we had when the problem hit us.

Get in contact with your system administrator. She/He should be able to provide any kind of monitoring you're looking for. (This option might only be available for the future, not the past)

how can we reproduce the problem and fix it for sure?

That can not be specifically answered unless you have found the cause. You can start monitoring and wait for it to occur or you can try to provoke this by pumping your server with thousands of requests at once.


I strongly suggest you start monitoring your server(s). This will give you a better view into which problems you might run. Both errors you describe look like networking problems. This can be that the server you're contacting (can be the local machine) is timing out (upstream timed out) or even refusing connection (connect failed).

You can then try to not allow too many connections to triggers this, so to shift the failing over to the user directly (instead of stressing your system too much) and/or if you could locate the cause, tweak the network settings to better deal with the requirements of your typical website usage. However it just can be possible that the server is technically not able to deal with the number / type of requests you're facing (hard limit). Monitoring will help you to find out.

You must log in to answer this question.