3

We have a staging Ubuntu server running on the Azure platform that just suddenly started timing out. We had created the server some days ago and it was working great. All of a sudden today while it was being used it just started returning 504s. We've restarted nginx and php-fpm and it's still not working.

We've tried removing the pass phrase from our private key like some other answers have said but it didn't fix the problem.

Any help or troubleshooting advice would be much appreciated. Also let me know if there's another config file you need to see.

Thanks in advance!

Our /var/log/nginx/error.log:

2014/09/12 14:11:33 [emerg] 3854#0: SSL_CTX_use_PrivateKey_file("/etc/ssl/ssl.key/myserver.key") failed (SSL: error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt  error:0906A065:PEM routines:PEM_do_header:bad decrypt error:140B0009:SSL routines:SSL_CTX_use_PrivateKey_file:PEM lib)
2014/09/12 14:12:49 [error] 3887#0: *1 upstream timed out (110: Connection timed out) while reading response header from upstream, client: {MY_IP}, server: {OUR_DOMAIN}, request: "GET / HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "{OUR_DOMAIN}"
2014/09/12 14:13:49 [error] 3887#0: *1 upstream timed out (110: Connection timed out) while reading response header from upstream, client: {MY_IP}, server: {OUR_DOMAIN}, request: "GET / HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "{OUR_DOMAIN}"
2014/09/12 14:20:14 [error] 3955#0: *1 upstream timed out (110: Connection timed out) while reading response header from upstream, client: {MY_IP}, server: {OUR_DOMAIN}, request: "GET / HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "{OUR_DOMAIN}"
Enter PEM pass phrase:
2014/09/12 14:20:55 [emerg] 4072#0: SSL_CTX_use_PrivateKey_file("/etc/ssl/ssl.key/myserver.key") failed (SSL: error:0906406D:PEM routines:PEM_def_callback:problems getting password error:0906A068:PEM routines:PEM_do_header:bad password read error:140B0009:SSL routines:SSL_CTX_use_PrivateKey_file:PEM lib)
2014/09/12 14:27:10 [alert] 4074#0: open socket #3 left in connection 3
2014/09/12 14:27:10 [alert] 4074#0: open socket #10 left in connection 4
2014/09/12 14:27:10 [alert] 4074#0: aborting
2014/09/12 14:40:08 [error] 4344#0: *5 upstream timed out (110: Connection timed out) while reading response header from upstream, client: {MY_IP}, server: {OUR_DOMAIN}, request: "GET / HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "{OUR_DOMAIN}"
2014/09/12 14:47:44 [error] 4344#0: *8 upstream timed out (110: Connection timed out) while reading response header from upstream, client: {MY_IP}, server: {OUR_DOMAIN}, request: "GET /{REQUESTED_PATH}/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "{OUR_DOMAIN}", referrer: "https://{OUR_DOMAIN}/{REQUESTED_PATH}/"

Our /var/log/nginx/access.log:

{MY_IP} - - [12/Sep/2014:14:11:24 +0000] "GET / HTTP/1.1" 504 593 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.120 Safari/537.36"
{MY_IP} - - [12/Sep/2014:14:13:49 +0000] "GET / HTTP/1.1" 504 593 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.120 Safari/537.36"
{MY_IP} - - [12/Sep/2014:14:18:50 +0000] "GET /{REQUESTED_PATH}/ HTTP/1.1" 499 0 "https://{OUR_DOMAIN}/{REQUESTED_PATH}/" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.103 Safari/537.36"
{MY_IP} - - [12/Sep/2014:14:26:18 +0000] "GET / HTTP/1.1" 499 0 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.120 Safari/537.36"
{MY_IP} - - [12/Sep/2014:14:30:07 +0000] "GET / HTTP/1.1" 499 0 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.120 Safari/537.36"

Our /etc/nginx/nginx.conf:

user www-data;
worker_processes 1;
pid /run/nginx.pid;

events {
    worker_connections 1024;
    # multi_accept on;
}

http {

    ##
    # Basic Settings
    ##

    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    keepalive_timeout 65;
    types_hash_max_size 2048;
    # server_tokens off;

    client_body_buffer_size 10K;
    client_header_buffer_size 1k;
    client_max_body_size 32m;
    large_client_header_buffers 2 1k;

    client_header_timeout 3000;
    client_body_timeout 3000;

    fastcgi_buffers 8 128k;
    fastcgi_buffer_size 128k;

    # server_names_hash_bucket_size 64;
    # server_name_in_redirect off;

    include /etc/nginx/mime.types;
    default_type application/octet-stream;

    ##
    # Logging Settings
    ##

    access_log /var/log/nginx/access.log;
    error_log /var/log/nginx/error.log;

    ##
    # Gzip Settings
    ##

    gzip on;
    gzip_disable "msie6";
    gzip_min_length  1000;
    gzip_proxied expired no-cache no-store private auth;
    gzip_comp_level 2;
    gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;

    # gzip_vary on;
    # gzip_proxied any;
    # gzip_comp_level 6;
    # gzip_buffers 16 8k;
    # gzip_http_version 1.1;
    # gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;

    ##
    # nginx-naxsi config
    ##
    # Uncomment it if you installed nginx-naxsi
    ##

    #include /etc/nginx/naxsi_core.rules;

    ##
    # nginx-passenger config
    ##
    # Uncomment it if you installed nginx-passenger
    ##

    #passenger_root /usr;
    #passenger_ruby /usr/bin/ruby;

    ##
    # Virtual Host Configs
    ##

    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-enabled/*;
}

Our /var/log/php5-fpm.log:

[12-Sep-2014 14:20:49] NOTICE: configuration file /etc/php5/fpm/php-fpm.conf test is successful

Our /etc/php5/fpm/php-fpm.conf:

;;;;;;;;;;;;;;;;;;;;;
; FPM Configuration ;
;;;;;;;;;;;;;;;;;;;;;

{COMMENTS}

;;;;;;;;;;;;;;;;;;
; Global Options ;
;;;;;;;;;;;;;;;;;;

[global]
{COMMENTS}
pid = /var/run/php5-fpm.pid

{COMMENTS}
error_log = /var/log/php5-fpm.log

{COMMENTS}

;;;;;;;;;;;;;;;;;;;;
; Pool Definitions ;
;;;;;;;;;;;;;;;;;;;;

{COMMENTS}
include=/etc/php5/fpm/pool.d/*.conf

Our /etc/php5/fpm/pool.d/www.conf:

{COMMENTS}
[www]

{COMMENTS}
user = www-data
group = www-data

{COMMENTS}
listen = /var/run/php5-fpm.sock

{COMMENTS}
listen.owner = www-data
listen.group = www-data
listen.mode = 0666

{COMMENTS}
pm = dynamic

{COMMENTS}
pm.max_children = 5

{COMMENTS}
pm.start_servers = 2

{COMMENTS}
pm.min_spare_servers = 1

{COMMENTS}
pm.max_spare_servers = 3

{COMMENTS}
request_terminate_timeout = 600

{COMMENTS}
chdir = /

{COMMENTS}

Software Versions:

  • nginx version: nginx/1.4.6 (Ubuntu)
  • Ubuntu 14.04.1 LTS (GNU/Linux 3.13.0-32-generix x86_64)
  • PHP 5.5.9-1ubuntu4.3 (fpm-fcgi)
  • Zend Engine v2.5.0 with Zend OPcache v7.0.3
Improve this question
1
  • Look for errors in the php-fpm layers (try to bounce it, check if /var/run/php5-fpm.sock exists, etc).
    – anon
    Sep 12, 2014 at 18:58

2 Answers 2

Reset to default
3

If this is a standalone server, try increasing the timeout in your php.ini. If it's not, if it's part of a network, there may be some type of backend communication issue. I had an issue just yesterday, where our db nodes (running percona) and api nodes were having trouble establishing connections. Once we cleared that up, and bounced nginx again, the problem went right away.

Improve this answer
1
  • Just had this problem and couldn't figure out what was going on with the web server, then discovered the database server wasn't responding.
    – DanH
    Feb 7, 2015 at 19:53
1

I ran into this issue today, and after 40 minutes of debugging found that php-fpm wasn't starting correctly. I think what had a happened was a pool worker hadn't stopped when fpm had crashed, and was holding onto the .sock file. Once I manually killed the pool workers php fpm started correctly and all started working well again.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .