36

After debugging for 6-hours - I am giving this up :|

We have a nginx+php-fpm+mysql in LAN with almost 100 wordpress (created and used by different designers/developers all working on test wordpres setup)

We are using nginx without any issues from long.

Today, all of a sudden - nginx started returning "504 Gateway Time-out" out of the blue...

I checked nginx error log for a virtual host...

2010/09/06 21:24:24 [error] 12909#0: *349 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.0.1, server: rahul286.rtcamp.info, request: "GET /favicon.ico HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "rahul286.rtcamp.info"
2010/09/06 21:25:11 [error] 12909#0: *349 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 192.168.0.1, server: rahul286.rtcamp.info, request: "GET /favicon.ico HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "rahul286.rtcamp.info"
2010/09/06 21:25:11 [error] 12909#0: *443 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 192.168.0.1, server: rahul286.rtcamp.info, request: "GET /info.php HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "rahul286.rtcamp.info"
2010/09/06 21:25:12 [error] 12909#0: *443 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.0.1, server: rahul286.rtcamp.info, request: "GET /favicon.ico HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "rahul286.rtcamp.info"
2010/09/06 22:08:32 [error] 12909#0: *1025 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.0.1, server: rahul286.rtcamp.info, request: "GET / HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "rahul286.rtcamp.info"
2010/09/06 22:09:33 [error] 12909#0: *1025 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.0.1, server: rahul286.rtcamp.info, request: "GET /favicon.ico HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "rahul286.rtcamp.info"
2010/09/06 22:09:40 [error] 12909#0: *1064 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 192.168.0.1, server: rahul286.rtcamp.info, request: "GET /info.php HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "rahul286.rtcamp.info"
2010/09/06 22:09:40 [error] 12909#0: *1064 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.0.1, server: rahul286.rtcamp.info, request: "GET /favicon.ico HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "rahul286.rtcamp.info"
2010/09/06 22:24:44 [error] 12909#0: *1313 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.0.1, server: rahul286.rtcamp.info, request: "GET / HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "rahul286.rtcamp.info"
2010/09/06 22:24:53 [error] 12909#0: *1313 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 192.168.0.1, server: rahul286.rtcamp.info, request: "GET /favicon.ico HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "rahul286.rtcamp.info"

As I run php-fpm on port 9000 via TCP mode, I ran "netstat | grep 9000" and noticed something unusual... (Pasting partial output here for ease of read)

tcp        9      0 localhost:9000          localhost:36094         CLOSE_WAIT  14269/php5-fpm  
tcp        0      0 localhost:46664         localhost:9000          FIN_WAIT2   -               
tcp     1257      0 localhost:9000          localhost:36135         CLOSE_WAIT  -               
tcp     1257      0 localhost:9000          localhost:36125         CLOSE_WAIT  -               
tcp        9      0 localhost:9000          localhost:36102         CLOSE_WAIT  14268/php5-fpm  
tcp        0      0 localhost:46662         localhost:9000          FIN_WAIT2   -               
tcp      745      0 localhost:9000          localhost:46644         CLOSE_WAIT  -               
tcp        0      0 localhost:46658         localhost:9000          FIN_WAIT2   -               
tcp     1265      0 localhost:9000          localhost:46607         CLOSE_WAIT  -               
tcp        0      0 localhost:46672         localhost:9000          ESTABLISHED 12909/nginx: worker
tcp     1257      0 localhost:9000          localhost:36119         CLOSE_WAIT  -               
tcp     1265      0 localhost:9000          localhost:46613         CLOSE_WAIT  -               
tcp        0      0 localhost:46646         localhost:9000          FIN_WAIT2   -               
tcp     1257      0 localhost:9000          localhost:36137         CLOSE_WAIT  -               
tcp        0      0 localhost:46670         localhost:9000          ESTABLISHED 12909/nginx: worker
tcp     1265      0 localhost:9000          localhost:46619         CLOSE_WAIT  -               
tcp     1336      0 localhost:9000          localhost:46668         ESTABLISHED -               
tcp        0      0 localhost:46648         localhost:9000          FIN_WAIT2   -               
tcp     1336      0 localhost:9000          localhost:46670         ESTABLISHED -               
tcp        9      0 localhost:9000          localhost:36108         CLOSE_WAIT  14274/php5-fpm  
tcp     1336      0 localhost:9000          localhost:46684         ESTABLISHED -               
tcp        0      0 localhost:46674         localhost:9000          ESTABLISHED 12909/nginx: worker
tcp     1336      0 localhost:9000          localhost:46666         ESTABLISHED -               
tcp     1257      0 localhost:9000          localhost:46648         CLOSE_WAIT  -               
tcp     1336      0 localhost:9000          localhost:46678         ESTABLISHED -               
tcp        0      0 localhost:46668         localhost:9000          ESTABLISHED 12909/nginx: wo             

There are plenty of "CLOSE_WAIT" & "FIN_WAIT2" pairs as highlighted below (in above output):

tcp     1337      0 localhost:9000          localhost:46680         CLOSE_WAIT  -               
tcp        0      0 localhost:46680         localhost:9000          FIN_WAIT2   -

Please note port 46680 in above.

I enabled mysql slow queries error log, but it didn't work.

As of now restarting php5-fpm every minute via a cronjob (see command below) keeping everything running "smoothly" but I hate patchwork and want to solve this...

1 * * * * service php5-fpm restart > /dev/null

I searched extensively on Google - got no help. As mentioned, this a test-server in LAN, CPU load is never crossed 0.10 and memory usage is also below 25% (System has 2GB RAM and ubuntu-server installed) So if you find its time-confusing to help me out, please atleast drop a hint.

Thanks in advance for help.

-Rahul

(note - this is reposting of - http://forum.nginx.org/read.php?11,127694)

Update: I found answer, which is posted below.

9 Answers 9

35

I found answer on my posting on the nginx forum - http://forum.nginx.org/read.php?2,127854

The answer, in my case, is to set:

request_terminate_timeout=30s

in php-fpm config (usually /etc/php5/fpm/php-fpm.conf)

Note, you can use values other than 30s also.

I used it to match my value in main php.ini file which is:

max_execution_time = 30

Thanks All. :-)

5
  • 5
    This config may also be found in www.conf file. Thanks for the answer though, this seems to have done the trick.
    – eddiemoya
    Aug 16, 2013 at 23:06
  • 2
    This is a pool-level directive, you will get an error message when trying to put it into the php-fpm.conf's [global] section. It works there only if you also have your pool configs in there. Also: request_terminate_timeout docs.
    – tanius
    Jan 30, 2015 at 22:19
  • If this is the correct answer which I REALLY NEEED then this Friday will be the best of 2015 yet.
    – Philip
    Feb 20, 2015 at 8:49
  • 2
    I found that putting request_terminate_timeout=30s into my php-fpm.conf file caused a (111 Connection Refused) error. When I moved it to my www.conf file it worked.
    – AJB
    Apr 20, 2016 at 0:17
  • On CentOS 7.2 when using php7, request_terminate_timeout is located in: /etc/php-fpm.d/www.conf
    – nadavkav
    Jan 1, 2017 at 13:45
19

Here how it resolved my issue:

make following changes to /etc/nginx/nginx.conf in http { section

proxy_connect_timeout  600s;
proxy_send_timeout  600s;
proxy_read_timeout  600s;
fastcgi_send_timeout 600s;
fastcgi_read_timeout 600s;

and then restart nginx

/etc/init.d/nginx restart

3
  • 2
    Yeah, that really doesn't look like it has anything to do with the problem the person asking the question has. Nov 19, 2012 at 11:39
  • 3
    but luckily that's what I needed :)
    – luchaninov
    Apr 2, 2013 at 20:30
  • This didn't fix my issue, but it allowed me to see the actual error instead of the timeout message, which led me to the actual issue.
    – Michael
    Sep 21, 2018 at 6:34
4

If you are using php 5.3, increase the backlog.

If you are using php 5.2, backport the patch to increase the backlog size from 128.

Also, use a unix socket rather than a TCP socket. unix:/tmp/php5-cgi.sock (or the relevant path)

4
  • I'd have to agree, especially with the unix socket usage.
    – Matt
    Sep 7, 2010 at 13:05
  • Thanks karmawhore for answer. I found a solution on nginx mailing list.
    – rahul286
    Sep 8, 2010 at 11:03
  • @rahul286 which answere? iam interested!
    – breiti
    Dec 15, 2011 at 19:50
  • @breiti see my anser below - serverfault.com/a/179136/17440
    – rahul286
    Dec 22, 2011 at 12:44
3

Great, thanks

request_terminate_timeout=30s

It works perfectly for me

but, I had to insert the line in this file : "/etc/php5/fpm/pool.d/www.conf" that is to say in the "Worker Section".

PHP 5.3.21-1 - Wordpress 3.5.1

http://php-fpm.org/wiki/Configuration_File

1
  • I had a combination of factors that end up causing the 502 error your recipe did the magic trick! thanks you so much! Mar 8, 2016 at 3:29
2

in my case (same nginx error message), some problematic php scripts are not ending to execute and waiting for something, resulting no more php5-fpm children for nginx to pick.

fix:

  1. add execution time limit other mentioned it this post. request_terminate_timeout=30s
  2. raise children number. and everything worked like a charm. pm.max_spare_servers=16 pm.min_spare_servers=2

now everything worked like a charm.

1
  • I had a long running external connection request in my php script. Look for those long running tasks and put a timeout for them. Dec 1, 2017 at 21:05
1

For me same issue has occured after removing rabbitmq from server. Nothing of the above wasnt usefull, reinstalling all php5 modules solved the problem. I had Debian 8.2 on that server. Hope will be helpful for someone.

0

This may also help folks:

Depending on what your setup is you should look at the fastcgi configuration params as well as php ... in my case (I am using apache2 + php5-fpm) and max_execution time is also dependent on how long the fastcgi module waits for response (-idle-timeout) ...

http://www.fastcgi.com/mod_fastcgi/docs/mod_fastcgi.html#FastCgiExternalServer

3
  • why use apache2?? I mean u can use nginx directly to interact with php5-fpm. No need to use Apache when u have nginx!
    – rahul286
    Mar 21, 2011 at 15:57
  • If your using nginx, if others are NOT using nginx hopefully this will help them out. :-) ... I came across this page searching for Apache2 + php5-fpm related question
    – farinspace
    Mar 21, 2011 at 18:27
  • ok. I thought you are using Nginx with Apache for PHP scripts like some pople used it in past.
    – rahul286
    Mar 22, 2011 at 16:05
0

I had the same problem and I solved it by completely removeing Apache:

yum remove httpd

After that I reccomend restaring both PHP and NGINX:

/etc/init.d/nginx restart
/etc/init.d/php-fpm restart
1
  • 1
    We did not have apache on our server then. Glad to know your case as it may help us in future.
    – rahul286
    Apr 12, 2012 at 12:44
0

In our case the problem was that php-fpm couldn't open a session on an NFS share , it seems like one of the web nodes locked the directory , after restarting the NFS and only allowed 1 node to access it the issue was resolved ,Thank God

You should always check your php-fpm logs , in my case the location was at /var/opt/remi/php72/log/php-fpm

you should also enable slow scripts logs for better debugging and check the www-slow.log file located in the logs directory

to make sure slowlog is working make sure the following settings present in www.conf

slowlog = /var/opt/remi/php72/log/php-fpm/www-slow.log

request_slowlog_timeout = 5s

this should log any script that exceeds 5 seconds and logs the location in the php code that makes the problem in /var/opt/remi/php72/log/php-fpm/www-slow.log , you can change the path to meet your environment

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .