Updated SSL certs and now rewrite conditions no longer working for apache2

Question

I'm running an Ubuntu 14.04 server with Apache2 v2.4.7.

Recently our site was getting errors from Chrome saying we needed to update our certs. I just completed that earlier this week, updating the SSL Certs and then restarting Apache. A couple of days layer, my client came back and mentioned that certain images and files weren't accessible anymore through their reporting server. The reporting server was using HTTPS to get to the images and files.

My conf file has this, which hasn't changed since I inherited this server:

<VirtualHost *:80>
    ServerName abc.example.com
    DocumentRoot /home/deployer/html/_abc/public
    RewriteEngine On

    RewriteCond %{HTTPS} off
    RewriteCond %{REQUEST_URI} !^/files/
    RewriteCond %{REQUEST_URI} !^/images/checksheets/
    RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI} [R,L]
</VirtualHost>

<VirtualHost *:443>
    ServerName abc.example.com
    DocumentRoot /home/deployer/html/_abc/public
    SSLEngine on
    SSLProtocol all -SSLv2 -SSLv3
    SSLCipherSuite          ECD....C3-SHA:!DSS
    SSLHonorCipherOrder     on
    SSLCompression          off
    SSLCertificateFile /etc/ssl/ssl_certificate.crt
    SSLCertificateKeyFile /etc/ssl/abc_staging.key
    SSLCertificateChainFile /etc/ssl/IntermediateCA.crt
    # HSTS (mod_headers is required) (15768000 seconds = 6 months)
    Header always set Strict-Transport-Security "max-age=15768000"
    <Directory /home/deployer/html/_abc/public>
        AllowOverride all
        Options FollowSymLinks
        # -MultiViews
        #   Order allow,deny
        Require all granted
    </Directory>

The only thing I've done on the server has literally been to update the ssl_certificate and IntermediateCAs and restart Apache, so I'm not sure what happened. The last time the SSL Cert was updated was probably about a year ago, and the reporting server's way of grabbing images/files hasn't changed. I also restart Apache fairly frequently and this had never been an issue previously.

On a whim, my client created a new report using http instead of https, and that seemed to work perfectly OK.

My question really is, what could have changed to cause the server to suddenly no longer accept https connections for images/files?

EDIT: These are only GET requests (there are no POST/DELETE actions).

Also, navigating to the image (or file) as https://abc.example.com/images/path/to/file.jpg with HTTPS completely works fine. But when the reporting server uses that same link, a blank page gets displayed.

Apache access logs will show the above link as a hit when navigating directly, but when the reporting server uses it, and I navigate to the report, Apache access logs show nothing at all.

Answer 1

On your <VirtualHost *:443> you enable HSTS. That tells the browser to internally rewrite every request for the domain to HTTPS. After this, there's no use for the RewriteCond %{REQUEST_URI} !^/files/ as everything should be rewritten unconditionally already, giving you the possibility to simplify your configuration:

<VirtualHost *:80>
    ServerName abc.example.com
    Redirect permanent / https://abc.example.com/
</VirtualHost>

The blank page problem doesn't seem a server side issue caused by your Apache at all. The fact that you can access the URL from another computer pretty much proves it.

First clear all caches on the "reporting server" and see whether it helps or not. There's no details how this "reporting server" works, but the blank page might for example be a result of not displaying a cached HTTP URL image on a HTTPS page due to a changed mixed content policy. If so, the consistency between the redirection and the HSTS policy might prevent such caching in the future.

Also, fix all http:// URLs on your content to https://.

Answer 2

Troubleshooting suggestions:

1. Please confirm your environment:

   • Ubuntu 14.04 server with Apache2 v2.4.7.
   • Your httpd.conf is configured to serve both HTTP (:80) and HTTPS (:443).
   • Your certificate configuration (posted above) is correct, and the files and (SSLCertificateFile, etc.) all exist and haven't been "corrupted".

2. Please confirm the problem description:

   • "Everything used to work" before you updated your SSL certificate, "the problem" has something to do with the new certificate.
   • All users who try to "get reports" via https (EXAMPLE https://apache-server.com/myreport) fail - they just see "white". Is this correct?
   • The same users are successfully able to get the same reports if they use http (EXAMPLE: http://apache-server.com/myreport) instead. Is this also correct?
   • Is there anything "special" about reports? For example, is it a web application?

3. I'm curious if ALL https is "broken", or just "some stuff" is inaccessible.

   Please create a test file (EXAMPLE: /var/www/html/myreport/hello.html) and verify whether or not you can read it with both http and https.

4. When you test, please use one (or both) of these tools on your PC to analyze the browser's attempt to connect (and thus help determine the specific point of failure):

   • Fiddler
   • Chrome Developer Tools/Network tab

5. Please post back what you find.

PS:

Here are some additional tips. In particular, "enable verbose logging":

https://www.futurehosting.com/blog/troubleshooting-the-most-common-apache-problems-youll-encounter/

PPS:

I would NOT disable http until you've resolved the problem:

• If nothing else, it's a temporary workaround for those folks who might need reports.
• Being able to compare http vs. https behavior will help troubleshooting.

Answer 3

We figured it out. While the Apache server was fine as is, apparently we previously had a specific DH param snippet saved in the certificate file. This snippet was added because the reporting server that asks for the images and files runs on Java 6. And Java 6 can only handle DH params of 1024bit.

So in my new ssl_certificate.crt, I added

-----BEGIN DH PARAMETERS-----
MIGHAoGBAP//////////yQ/aoiFowjTExmKLgNwc0SkCTgiKZ8x0Agu+pjsTmyJR
Sgh5jjQE3e+VGbPNOkMbMCsKbfJfFDdP4TVtbVHCReSFtXZiXn7G9ExC6aY37WsL
/1y29Aa37e44a/taiZ+lrp8kEXxLH+ZJKGZR7OZTgf//////////AgEC
-----END DH PARAMETERS----- 

from http://httpd.apache.org/docs/current/ssl/ssl_faq.html#javadh

Unfortunately that downgrades our certificate from A+ rating to a B, but I'm not allowed to upgrade the Java server from 6 to whatever newer one.

Once I added that snippet to my ssl cert, the reporting server was able to access the images over https!