I'm getting error: SSL3_GET_RECORD:decryption failed or bad record mac

Question

I have my own server (where I'm running Apache/2.4.27), and today I realized that from (Brave and Google Chrome - different computers) I'm getting from my websites this error;

This site can’t provide a secure connection

mywebsite.com sent an invalid response.
ERR_SSL_PROTOCOL_ERROR

And the strange thing is that I'm getting this error every fifth click on my website.

From my conf file:

SSLEngine on
SSLCertificateFile /etc/letsencrypt/live/mywebsite/cert.pem
SSLCertificateKeyFile /etc/letsencrypt/live/mywebsite/privkey.pem
Include /etc/letsencrypt/options-ssl-apache.conf
SSLCertificateChainFile /etc/letsencrypt/live/mywebsite/chain.pem
SSLCompression off

from options-ssl-apache.conf;

SSLProtocol             all -SSLv2 -SSLv3
SSLCipherSuite          EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH
SSLHonorCipherOrder     on
SSLCompression          off

I have checked log file from website but nothing, also nothing here; /var/log/apache2/error.log

I'm trying to figure out what is causing this error, any ideas where can I find more info or even better, how to solve this problem?

EDIT:

If I try openssl s_client -connect mywebsite.com:443, it will return:

I'm using: OpenSSL 1.1.0f

CONNECTED(00000003)

...

3073276480:error:1408F119:SSL routines:ssl3_get_record:decryption failed or bad record mac:../ssl/record/ssl3_record.c:469:

ANOTHER EDIT:

As @quadruplebucky suggested I changed options-ssl-apache.conf into:

SSLProtocol             all -SSLv2 -SSLv3
SSLCipherSuite           HIGH:MEDIUM:!aNULL:!MD5:!SSLv3:!SSLv2:!TLSv1
SSLHonorCipherOrder     on
SSLCompression          off

#SSLSessionTickets       off

I also tried to add SSLProtocol all -SSLv2 -SSLv3 into my virtualhost conf file, and in a same time I did change couple of things here; /etc/apache2/mods-available/ssl.conf

#SSLCipherSuite HIGH:!aNULL
SSLCipherSuite HIGH:MEDIUM:!aNULL:!MD5:!SSLv3:!SSLv2:!TLSv1

SSLHonorCipherOrder on

#   The protocols to enable.
#   Available values: all, SSLv3, TLSv1, TLSv1.1, TLSv1.2
#   SSL v2  is no longer supported
SSLProtocol all -SSLv2 -SSLv3

EDIT:

After changing LogLevel into Info it returns:

[Sat Jul 08 13:34:53.374307 2017] [ssl:info] [pid 8710] [client] AH02008: SSL library error 1 in handshake (server mywebsite:443)
[Sat Jul 08 13:34:53.374717 2017] [ssl:info] [pid 8710] SSL Library Error: error:140940F4:SSL routines:ssl3_read_bytes:unexpected message
[Sat Jul 08 13:34:53.374750 2017] [ssl:info] [pid 8710] [client] AH01998: Connection closed to child 1 with abortive shutdown (server mywebsite:443)

EDIT:

If I run with option -crlf, like this:

openssl s_client -crlf -connect mywebsite:443

I'm getting no error?

One more thing if I change LogLevel to debug, before that error I'm getting this:

[Tue Jul 11 23:00:38.641568 2017] [core:debug] [pid 26561] protocol.c(1273): [client 188.64.25.162:23165] AH00566: request failed: malformed request line
[Tue Jul 11 23:00:38.641634 2017] [headers:debug] [pid 26561] mod_headers.c(900): AH01503: headers: ap_headers_error_filter()

So after this that same error will happen:

SSL Library Error: error:140940F4:SSL routines:ssl3_read_bytes:unexpected message

openssl version
OpenSSL 1.1.0f  25 May 2017

Answer 1

You can't tell from that error if your server is negotiating SSLv3 or TLSv1 (you might want to have a look at this question on Unix & Linux and make sure it's disabled everywhere in apache...) --- the 1.1.0f source code here on GitHub deliberately blurs the two...

   if (enc_err < 0) {
    /*
     * A separate 'decryption_failed' alert was introduced with TLS 1.0,
     * SSL 3.0 only has 'bad_record_mac'.  But unless a decryption
     * failure is directly visible from the ciphertext anyway, we should
     * not reveal which kind of error occurred -- this might become
     * visible to an attacker (e.g. via a logfile)
     */
    al = SSL_AD_BAD_RECORD_MAC;
    SSLerr(SSL_F_SSL3_GET_RECORD,
           SSL_R_DECRYPTION_FAILED_OR_BAD_RECORD_MAC);
    goto f_err;
}

So you might want to reorder your cipher suite:

SSLCipherSuite HIGH:MEDIUM:!aNULL:!MD5:!SSLv3:!SSLv2:!TLSv1

This post on askubuntu about the POODLE vulnerability has an excellent list of resources for SSL inspection and plumbing.

The Mozilla config generator is an excellent public service.

The "getting this error every fifth click" comment is a little strange. Do you mean clicks, or every fifth line is a bad request in the logs? Try starting apache single-threaded (the -X flag) and see if that does the same thing...or maybe setting SSLSessionTickets off.

My thinking here is to eliminate threading and session/cache coherency/consistency as source of trouble. Running apache single-threaded (starting it with the -X flag) is one way to accomplish this, another way is to set MaxClients=1 (at least with the MPM model). Session tickets have been a source of trouble in the past with TLSv1.2 and they are enabled by default, this is the reasoning behind SSLSessionTickets off (note this is part of the SSL "Server Hello" message, not a session cookie or similar). The 'every fifth click' error still bothers me - I can't help but notice that most browsers will pipeline four resource requests in a single one...) and open a new connection (a new ssl handshake, etc...) for the fifth...without a packet capture it's hard to say what's actually going on.

It would seem that you've eliminated cipher negotiation as a source of error (you can duplicate the error condition under much more restrictive cipher specs, unless I'm mistaken). I would be curious to know if you can trigger the error by renegotiating SSL (just for kicks, say): openssl s_client -connect server:443 and then type 'R', see what the logs say.
Also see if session caching is working with the -reconnect option to s_client.
Something must be different about the receiving contexts for the SSL requests, and it seems the best way to figure that out (short of a byte-by-byte inspection of what's going over the wire, which might be tough to anonymize) is to severely limit the size of what's listening (that is, the number of listeners).

Other debugging tools I'd try (assuming posting packet captures is out of the question....)
- ssltap (in libnss3-tools on ubuntu)
- cipherscan
- sslscan

UPDATE
Poking at this through ssltap looks an awful lot like OpenSSL bug #3712 resurfaced (key renegotiation during read/write, basically). Looking for a decent workaround that won't kill performance. Fun stuff!

Answer 2

I've seen this before, had this before in fact.

The answer in my case ended up being extremely subtle.

The network adapter enabled TCP Segment Offloading, which due to a bug of some form was mangling (or truncating, cant remember) the last few bytes of some messages -- which was subsequently causing the MAC on the SSL records to fail.

It was a virtual machine on VMWare.

I would try disabling TSO/GSO/GRO in your environment and seeing if the problem goes away.

ethtool -K eth0 tso off gro off gso off ufo off

Answer 3

There is some weirdness with OpenSSL and multithreading. What MPM do you use? If this is multithreading-related the "prefork" should be safe while "worker" and "event" might be affected.

If your load-profile allows it maybe you can try to switch over to prefork and see if the issue persists.

Answer 4

Having similar error sometimes when I work with my smartphone as a wifi hubspot. Rebooting the connexion solve the problem.

Answer 5

First make sure chrome is updated. Older versions give problems with certain ciphers. Had this issue with chromium myself with common sites like amazon, etc.

Second, the advice to forbid protocols in "cipher list" you followed is a very bad idea because it won't forbid protocols, it will forbid most ciphers including those that work "since" SSLv3 (but does not mean you are enabling SSL3 if you allow SSLv3 ciphers), use a more generioc list provided by Mozilla SSL config generator for compatibility (note SSLv2 no longer exists or is supported in httpd or openssl so no reason to forbid it explicitately any more) and perhaps your previous combination was too strict, try this one:

SSLProtocol             all -SSLv3
SSLCipherSuite          ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS

If you still have issues, enable debugging for SSL and see what httpd has to say (only send 1 try or two and disable this logging, this is too noisy):

LogLevel ssl:trace3

Sidenotes: You can also go ahead and remove SSLCertificateChainFile since that directive is deprecated in 2.4. You can add the chain the SSLCertificateFile and sort all the certificates from leaf to root, or even change SSLCertificateChainFile to SSLCACertificateFile (although this one is mainly used for SSL auth).

You should also add (if you haven't yet) to add:

SSLRandomSeed startup file:/dev/urandom 2048
SSLRandomSeed connect file:/dev/urandom 2048
SSLSessionCache shmcb:/path/to/log/ssl_gcache_data(512000)

---

Edit: Following our conversation let's resort to check openssl installation and/or if it really is the same version your httpd uses:

Issue this command and let's see what it says: openssl ciphers -v 'ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS'

Also to make sure the openssl version is the correct one run this:

openssl version