Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

After debugging for 6-hours - I am giving this up :|

We have a nginx+php-fpm+mysql in LAN with almost 100 wordpress (created and used by different designers/developers all working on test wordpres setup)

We are using nginx without any issues from long.

Today, all of a sudden - nginx started returning "504 Gateway Time-out" out of the blue...

I checked nginx error log for a virtual host...

2010/09/06 21:24:24 [error] 12909#0: *349 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.0.1, server: rahul286.rtcamp.info, request: "GET /favicon.ico HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "rahul286.rtcamp.info"
2010/09/06 21:25:11 [error] 12909#0: *349 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 192.168.0.1, server: rahul286.rtcamp.info, request: "GET /favicon.ico HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "rahul286.rtcamp.info"
2010/09/06 21:25:11 [error] 12909#0: *443 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 192.168.0.1, server: rahul286.rtcamp.info, request: "GET /info.php HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "rahul286.rtcamp.info"
2010/09/06 21:25:12 [error] 12909#0: *443 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.0.1, server: rahul286.rtcamp.info, request: "GET /favicon.ico HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "rahul286.rtcamp.info"
2010/09/06 22:08:32 [error] 12909#0: *1025 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.0.1, server: rahul286.rtcamp.info, request: "GET / HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "rahul286.rtcamp.info"
2010/09/06 22:09:33 [error] 12909#0: *1025 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.0.1, server: rahul286.rtcamp.info, request: "GET /favicon.ico HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "rahul286.rtcamp.info"
2010/09/06 22:09:40 [error] 12909#0: *1064 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 192.168.0.1, server: rahul286.rtcamp.info, request: "GET /info.php HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "rahul286.rtcamp.info"
2010/09/06 22:09:40 [error] 12909#0: *1064 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.0.1, server: rahul286.rtcamp.info, request: "GET /favicon.ico HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "rahul286.rtcamp.info"
2010/09/06 22:24:44 [error] 12909#0: *1313 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.0.1, server: rahul286.rtcamp.info, request: "GET / HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "rahul286.rtcamp.info"
2010/09/06 22:24:53 [error] 12909#0: *1313 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 192.168.0.1, server: rahul286.rtcamp.info, request: "GET /favicon.ico HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "rahul286.rtcamp.info"

As I run php-fpm on port 9000 via TCP mode, I ran "netstat | grep 9000" and noticed something unusual... (Pasting partial output here for ease of read)

tcp        9      0 localhost:9000          localhost:36094         CLOSE_WAIT  14269/php5-fpm  
tcp        0      0 localhost:46664         localhost:9000          FIN_WAIT2   -               
tcp     1257      0 localhost:9000          localhost:36135         CLOSE_WAIT  -               
tcp     1257      0 localhost:9000          localhost:36125         CLOSE_WAIT  -               
tcp        9      0 localhost:9000          localhost:36102         CLOSE_WAIT  14268/php5-fpm  
tcp        0      0 localhost:46662         localhost:9000          FIN_WAIT2   -               
tcp      745      0 localhost:9000          localhost:46644         CLOSE_WAIT  -               
tcp        0      0 localhost:46658         localhost:9000          FIN_WAIT2   -               
tcp     1265      0 localhost:9000          localhost:46607         CLOSE_WAIT  -               
tcp        0      0 localhost:46672         localhost:9000          ESTABLISHED 12909/nginx: worker
tcp     1257      0 localhost:9000          localhost:36119         CLOSE_WAIT  -               
tcp     1265      0 localhost:9000          localhost:46613         CLOSE_WAIT  -               
tcp        0      0 localhost:46646         localhost:9000          FIN_WAIT2   -               
tcp     1257      0 localhost:9000          localhost:36137         CLOSE_WAIT  -               
tcp        0      0 localhost:46670         localhost:9000          ESTABLISHED 12909/nginx: worker
tcp     1265      0 localhost:9000          localhost:46619         CLOSE_WAIT  -               
tcp     1336      0 localhost:9000          localhost:46668         ESTABLISHED -               
tcp        0      0 localhost:46648         localhost:9000          FIN_WAIT2   -               
tcp     1336      0 localhost:9000          localhost:46670         ESTABLISHED -               
tcp        9      0 localhost:9000          localhost:36108         CLOSE_WAIT  14274/php5-fpm  
tcp     1336      0 localhost:9000          localhost:46684         ESTABLISHED -               
tcp        0      0 localhost:46674         localhost:9000          ESTABLISHED 12909/nginx: worker
tcp     1336      0 localhost:9000          localhost:46666         ESTABLISHED -               
tcp     1257      0 localhost:9000          localhost:46648         CLOSE_WAIT  -               
tcp     1336      0 localhost:9000          localhost:46678         ESTABLISHED -               
tcp        0      0 localhost:46668         localhost:9000          ESTABLISHED 12909/nginx: wo             

There are plenty of "CLOSE_WAIT" & "FIN_WAIT2" pairs as highlighted below (in above output):

tcp     1337      0 localhost:9000          localhost:46680         CLOSE_WAIT  -               
tcp        0      0 localhost:46680         localhost:9000          FIN_WAIT2   -

Please note port 46680 in above.

I enabled mysql slow queries error log, but it didn't work.

As of now restarting php5-fpm every minute via a cronjob (see command below) keeping everything running "smoothly" but I hate patchwork and want to solve this...

1 * * * * service php5-fpm restart > /dev/null

I searched extensively on Google - got no help. As mentioned, this a test-server in LAN, CPU load is never crossed 0.10 and memory usage is also below 25% (System has 2GB RAM and ubuntu-server installed) So if you find its time-confusing to help me out, please atleast drop a hint.

Thanks in advance for help.

-Rahul

(note - this is reposting of - http://forum.nginx.org/read.php?11,127694)

Update: I found answer, which is posted below.

share|improve this question
add comment

7 Answers

up vote 12 down vote accepted

I found answer on my posting on nginx forum - http://forum.nginx.org/read.php?2,127854

The answer, in my case, is to set:

request_terminate_timeout=30s

in php-fpm config.

Note, you can use values other than 30s also.

I used it to match my value in main php.ini file which is:

max_execution_time = 30

Thanks All. :-)

share|improve this answer
1  
This config may also be found in www.conf file. Thanks for the answer though, this seems to have done the trick. –  eddiemoya Aug 16 '13 at 23:06
    
It is not working for me –  Nanhe Kumar Sep 25 '13 at 22:04
add comment

Here how it resolved my issue:

make following changes to /etc/nginx/nginx.conf in http { section

proxy_connect_timeout  600s;
proxy_send_timeout  600s;
proxy_read_timeout  600s;
fastcgi_send_timeout 600s;
fastcgi_read_timeout 600s;

and then restart nginx

/etc/init.d/nginx restart

share|improve this answer
    
Yeah, that really doesn't look like it has anything to do with the problem the person asking the question has. –  HopelessN00b Nov 19 '12 at 11:39
1  
but luckily that's what I needed :) –  how Apr 2 '13 at 20:30
    
This fixed the issue for me also, thanks. –  Shane Grant Sep 18 '13 at 3:29
add comment

If you are using php 5.3, increase the backlog.

If you are using php 5.2, backport the patch to increase the backlog size from 128.

Also, use a unix socket rather than a TCP socket. unix:/tmp/php5-cgi.sock (or the relevant path)

share|improve this answer
    
I'd have to agree, especially with the unix socket usage. –  Matt Sep 7 '10 at 13:05
    
Thanks karmawhore for answer. I found a solution on nginx mailing list. –  rahul286 Sep 8 '10 at 11:03
    
@rahul286 which answere? iam interested! –  breiti Dec 15 '11 at 19:50
    
@breiti see my anser below - serverfault.com/a/179136/17440 –  rahul286 Dec 22 '11 at 12:44
add comment

I had the same problem and I solved it by completely removeing Apache:

yum remove httpd

After that I reccomend restaring both PHP and NGINX:

/etc/init.d/nginx restart
/etc/init.d/php-fpm restart
share|improve this answer
1  
We did not have apache on our server then. Glad to know your case as it may help us in future. –  rahul286 Apr 12 '12 at 12:44
add comment

Great, thanks

request_terminate_timeout=30s

It works perfectly for me

but, I had to insert the line in this file : "/etc/php5/fpm/pool.d/www.conf" that is to say in the "Worker Section".

PHP 5.3.21-1 - Wordpress 3.5.1

http://php-fpm.org/wiki/Configuration_File

share|improve this answer
add comment

in my case (same nginx error message), some problematic php scripts are not ending to execute and waiting for something, resulting no more php5-fpm children for nginx to pick.

fix:

  1. add execution time limit other mentioned it this post. request_terminate_timeout=30s
  2. raise children number. and everything worked like a charm. pm.max_spare_servers=16 pm.min_spare_servers=2

now everything worked like a charm.

share|improve this answer
add comment

This may also help folks:

Depending on what your setup is you should look at the fastcgi configuration params as well as php ... in my case (I am using apache2 + php5-fpm) and max_execution time is also dependent on how long the fastcgi module waits for response (-idle-timeout) ...

http://www.fastcgi.com/mod_fastcgi/docs/mod_fastcgi.html#FastCgiExternalServer

share|improve this answer
    
why use apache2?? I mean u can use nginx directly to interact with php5-fpm. No need to use Apache when u have nginx! –  rahul286 Mar 21 '11 at 15:57
    
If your using nginx, if others are NOT using nginx hopefully this will help them out. :-) ... I came across this page searching for Apache2 + php5-fpm related question –  farinspace Mar 21 '11 at 18:27
    
ok. I thought you are using Nginx with Apache for PHP scripts like some pople used it in past. –  rahul286 Mar 22 '11 at 16:05
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.