9

We've got nginx running on Ubuntu Trusty. It serves several websites over https, running on one ip address.

Randomly, although it seems slightly related to work load, sometimes single requests turn up on the wrong vhost. This leads to requests on lustrum.thalia.nu being served by thalia.nu and vice-versa. This then gives nasty error pages as users suddenly end up on a different web site. When you press F5, users then end up on the original target again.

It does not seem browser or operating system related. It's been confirmed to happen on Firefox (Linux, Windows, Mac), Edge (Windows) and Chrome (Linux, Windows, Android) and Safari (iOS).

The issue appears to occur more frequently when the system is put under load, suggesting some sort of race condition.

lustrum.thalia.nu

server {
        server_name lustrum.thalia.nu;

        listen 443 ssl;

        ssl on;
        ssl_certificate /etc/nginx/certs/lustrum.thalia.nu.crt;
        ssl_certificate_key /etc/nginx/certs/lustrum.thalia.nu.key;

        add_header Strict-Transport-Security "max-age=63072000; preload";

        root /var/www/thalia-lustrum/public_html;

        location / {
                index index.php;
                try_files $uri $uri/ /index.php?$args;
        }

        # Add trailing slash to */wp-admin requests.
        rewrite /wp-admin$ $scheme://$host$uri/ permanent;

        # Pass all .php files onto a php-fpm/php-fcgi server.
        location ~ [^/]\.php(/|$) {
                include         /etc/nginx/fastcgi_params;

                fastcgi_split_path_info ^(.+?\.php)(/.*)$;

                if (!-f $document_root$fastcgi_script_name) {
                        return 404;
                }

                fastcgi_pass    unix:/var/run/php5-fpm-thalia-lustrum.sock;
                fastcgi_index   index.php;
                fastcgi_param   SCRIPT_FILENAME  /public_html$fastcgi_script_name;
        }
}

thalia.nu

server {
        server_name thalia.nu;    
        listen 443 ssl;

        ssl on;
        ssl_certificate /etc/nginx/certs/www.thalia.nu.crt;
        ssl_certificate_key /etc/nginx/certs/www.thalia.nu.key;

        add_header Strict-Transport-Security "max-age=63072000; preload";

        root /var/www/thalia/public_html;

        location / {
                try_files $uri $uri/ /index.php/$request_uri;
                index index.php index.html index.htm;
        }

        location ~ \.php($|/) {
                include         /etc/nginx/fastcgi_params;
                set  $script     $uri;
                set  $path_info  "";
                if ($uri ~ "^(.+\.php)(/.+)") {
                                set  $script     $1;
                                set  $path_info  $2;
                }
                fastcgi_read_timeout    120;
                fastcgi_pass    unix:/var/run/php5-fpm-thalia-www.sock;
                fastcgi_index   index.php;
                fastcgi_param   SCRIPT_FILENAME  /public_html$fastcgi_script_name;
        }
}

As you can see, we're running different PHP5-FPM pools for these two domains. These pools are chrooted to different folders and run as different users. PHP-FPM's configuration are otherwise fairly standard as far as I can tell.

We've tried both nginx 1.4.6-ubuntu3 and nginx 1.8.0-1+trusty.

Log telemetry

266.266.266.266 - - [25/Nov/2015:09:24:40 +0100] "GET /committees/175 HTTP/1.1" 302 5 "https://thalia.nu/committees" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:42.0) Gecko/20100101 Firefox/42.0" Host: "thalia.nu" Location: "https://thalia.nu/index.php//committees/wp-admin/setup-config.php"

In this line you can see that the request for the page /committees suddenly gets redirected to wp-admin. This appears that the request for /committees got handled by the thalia-lustrum PHP-fpm pool...

DNS zone file

We don't see how this can possibly be relevant, but...

;; MX Records
thalia.nu.    300    IN    MX    20    relay.transip.nl.
thalia.nu.    300    IN    MX    10    ivo.thalia.nu.

;; TXT Records
thalia.nu.    300    IN    TXT    "v=spf1 a mx a:mulgore.hexon-is.nl a:moonray.hexon-is.nl a:fred.thalia.nu a:ivo.thalia.nu ~all"

;; SPF Records (Sender Policy Framework)
thalia.nu.    300    IN    SPF    "v=spf1 a mx a:mulgore.hexon-is.nl a:moonray.hexon-is.nl a:fred.thalia.nu a:ivo.thalia.nu ~all"

;; CNAME Records
lustrum.thalia.nu.    300    IN    CNAME    thalia.nu.

;; A Records (IPv4 addresses)
thalia.nu.    300    IN    A    131.174.31.8
www.thalia.nu.    300    IN    A    131.174.31.8
ivo.thalia.nu.    300    IN    A    131.174.31.8
  • 1
    Plesse check your DNS settings for the domains. – Diamant Nov 18 '15 at 20:39
  • 1
    @bangal they are an A and a CNAME record, pointing to the same IP. I do not see how this is related, though; these resolve just fine, and it seems unlikely that a DNS issue would manifest so inconsistently. – Joost Nov 18 '15 at 22:23
  • 2
    @ThomWiggers, can you add to your log file the content of the Host: http header and user agent? See here for how: serverfault.com/questions/636790/…. Actually i tried makeing some requests to your websites but culdn't reproduce your problem. What client are you using for reproducing this? – Fredi Nov 24 '15 at 13:15
  • 3
    Is the fact that I just got "Third party content not installed" or something because you're working on it, or did I end up at another PHP pool or something (triggering the same bug)? I also got a brief error about config.php not found. – Halfgaar Nov 25 '15 at 8:26
  • 2
    @kasperd serverfault.com/questions/737349/…. It does appear to only affect PHP scripts. – Thom Wiggers Nov 25 '15 at 20:17
4
+50

After hours of debugging this issue we've finally been able to trace it to the cause. It appears the cause isn't nginx, but PHP-fpm. We're running php5-fpm version 5.5.9-1ubuntu4.14. It appears that when forking new workers, something sometimes goes wrong and the workers run (part?) of the code of different workers.

Our solution was to copy /etc/php5/fpm/php5-fpm.conf to different copies with their own pool.d folders, then to copy /etc/init.d/php5-fpm to launch with the new config file (also creating files in /etc/init/). This means we now have a php5-fpm process manager per pool. Having seperate chroots and sockets don't appear to keep things separate enough.

  • Note that it is currently unclear if this is an issue in our configuration or in (this version of) php5-fpm, although the latter does not seem likely given the lack of similar reports. If we end up finding the reason why this problem occurs, this answer will be updated. – Joost Nov 26 '15 at 10:42
1

I am facing the same issue but on Debian with Apache2.4.25 and PHP7.1-FPM. Here is a way to separate processes https://ma.ttias.be/a-better-way-to-run-php-fpm/

For those like me who might find this solution too heavy for small websites, add php_admin_value[opcache.revalidate_freq] = 0 at the end of the php-fpm pool configuration file. However, that may have a serious impact on performances...

Here is the official bug report: https://bugs.php.net/bug.php?id=67141

0

Does Nginx support SNI? You can run nginx -V and should see something like TLS SNI support enabled. If you don't, that may be why because the hostname is sent after the handshake and I'm assuming you have a wildcard certificate for *.thalia.nu

  • Of course it does, without SNI this would go wrong 100% of the time instead of very occasionally. (and I've also checked this, it is definitely enabled) – Thom Wiggers Nov 24 '15 at 11:02
  • FWIW, note that we do not serve a wildcard certificate, but use individual certificates for the separate subdomains. This is included in the configuration listed in the question. – Joost Nov 24 '15 at 11:12
  • ..although the lustrum.thalia.nu certificate is also valid for Thalia.nu – Thom Wiggers Nov 24 '15 at 11:17
  • Can you try adding the includeSubDomains parameter like this? add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"; – Mugurel Nov 24 '15 at 11:18
  • @ThomWiggers If the certificate is valid for multiple domains it is possible to support multiple domains on a single IP without the need for SNI. – kasperd Nov 24 '15 at 11:42
-1

It seems that the certificate is not right: firefox is telling me that it is issued for www.thalia.nu, not thalia.nu.

This is IMHO what is causing trouble. Try with another certificate or try activating HTTP connections without SSL.

  • We cannot reproduce that. The certificate served at www.thalia.nu and thalia.nu include both domains, with and without www. What Firefox version are you using, and on what platform? – Joost Nov 25 '15 at 19:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.