0

I switched a busy site to TLS-everywhere and HTTP/2 last week. Performance hasn't improved as much as hoped. In fact, depending on how you measure it, you can argue it has degraded.

There is plenty of scope to optimize the server further, at OS level TCP tweaks, TLS config in the web stack etc. To ensure I'm not overlooking any points for improvement, I'm posting my line of thought here for others to chime in.

Currently, the SSL conf shows up like this. Various parties control various elements of this, I don't have direct control over all of it. Domain name censored.

$ openssl s_client -state -CAfile Documents/Thawte\ Server\ CA.cer -connect xxxxxx.tld:443
CONNECTED(00000003)
SSL_connect:before/connect initialization
SSL_connect:SSLv2/v3 write client hello A
SSL_connect:SSLv3 read server hello A
depth=3 /C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
verify error:num=19:self signed certificate in certificate chain
verify return:0
SSL_connect:SSLv3 read server certificate A
SSL_connect:SSLv3 read server key exchange A
SSL_connect:SSLv3 read server done A
SSL_connect:SSLv3 write client key exchange A
SSL_connect:SSLv3 write change cipher spec A
SSL_connect:SSLv3 write finished A
SSL_connect:SSLv3 flush data
SSL_connect:SSLv3 read finished A
---
Certificate chain
 0 s:/C=xxx
   i:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Organization Validation Secure Server CA
 1 s:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
   i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
 2 s:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Certification Authority
   i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
 3 s:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Organization Validation Secure Server CA
   i:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Certification Authority
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIF1----shZZU
y1kFXCCRo78=
-----END CERTIFICATE-----
subject=/C=xxx
issuer=/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Organization Validation Secure Server CA
---
No client certificate CA names sent
---
SSL handshake has read 6240 bytes and written 328 bytes
---
New, TLSv1/SSLv3, Cipher is DHE-RSA-AES128-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : DHE-RSA-AES128-SHA
    Session-ID: 09464B1CF8972B08267C33BF255FDA74216728E3599B24EAE02B13642C268483
    Session-ID-ctx: 
    Master-Key: ACC3529AC7C80162797397D4E3041D1720EE4BA6A8BA5D2C200F8B48ADD6960CA9E48D5FF975472A9E8B43B7023CA4E4
    Key-Arg   : None
    Start Time: 1457708322
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---
SSL3 alert read:warning:close notify
closed
SSL3 alert write:warning:close notify

Problems I see:

verify error:num=19:self signed certificate in certificate chain

I want the chain as short as possible, in the right order (it isn't according to ssllabs.com) and there shouldn't be any need for self-signed certificates in the chain. Edit: I see this is actually the CA root certificate, which is already in the browser's trust store. This should be removed anyhow.

SSL_connect:SSLv3 read server certificate A SSL_connect:SSLv3 read server key exchange A SSL_connect:SSLv3 read server done A SSL_connect:SSLv3 write client key exchange A SSL_connect:SSLv3 write change cipher spec A SSL_connect:SSLv3 write finished A SSL_connect:SSLv3 flush data SSL_connect:SSLv3 read finished A

This handshake sequence seems long to me. I've seen fewer back-and-forths on other configurations. My desire is to keep round trips to a minimum.

Certificate Chain 0,1,2,3 - made up of 4 (Comodo) certificates, it is over 5.5Kb and hence may not fit in the TCP congestion window size (not verified yet). Probably the cause for extra round trips (?).

Perhaps it's worth switching to a CA which offers shorter chains. Any suggestions?

Cipher : DHE-RSA-AES128-SHA

Weak Diffy Helman cipher. SSLLabs adds: "Uses common DH primes Yes Replace with custom DH parameters if possible"

My questions hence are:

  • Which ways can I shorten the certificate chain?
  • Which ways can I reduce the handshake download size?
  • Which ways can I reduce the handshake round trips?
  • Which CA offers shorter chains than Comodo?
  • Do you see any other problem points in this output? Or what else should I test?
  • Which ways can we optimize TCP config on Ubuntu to maximize TLS performance / network throughput?
  • What is the best way to pick cipher suite order for performance? I ran $ openssl speed but not sure how to interpret and action this data.

Your insights would be much appreciated.

3

Lets sort the certificate chain issues first. This should take care of the first 4 points.

You will have been sent a certificate from Comodo and you will also need some more of the certificate chain. A list of the certificates can be found here. You will need to inspect the certificate to find out what CA it was signed with.

Once you have identified the CA download the bundle and place it in the same directory as the certificate you were sent. You will then need to combine them using the following command

cat name_of_file_you_were_sent name_of_bundle_file > www.example.com.crt

You may notice that the final CA is "missing" from the bundle, this won't stop you passing this part of the test on ssllabs.

You will then need to reference this file along with the private key file in the nginx config (Which you obviously have done but for completeness) using these two config lines

ssl_certificate     www.example.com.crt;
ssl_certificate_key www.example.com.key;

Test nginx config and restart if everything is ok


As for the rest of the configuration.

The best site I have found for generating an SSL config for nginx is Mozilla Security I've not included an example as there are many on serverfault alone

Depending on your site traffic you will either want to use intermediate or Modern (Hopefully you have no need to use old and if you do you may want to deal with that, but that's a different question)

You shouldn't just copy and paste those configs without understanding what the settings will do. HSTS for example can have long term impact on site visitors.

While in general these configuration are "safe" they can be optimised a little more. Specifically

ssl_buffer_size 16k; #Better if you have video
ssl_buffer_size 4k; #Better for quicker first byte.

Of course there could be some other reasons for why the switch to http/2 did not increase your site performance.

For example http/2 opens many more connections to the server. Did you account for this and adjust worker_processes and worker_connections but the above deal with anything directly related to the SSL config parts of nginx.

  • Thanks, I'll check your links. I'm aware of those buffer size settings, as well as size settings for ssl_session_cache and ssl_session_timeout. Rather than trying and guessing, testing, then going back, I'm keen to properly understand these values, and coming up with meaningful attempts. Hence looking at certificate chain size vs TCP packet size etc., obtaining true numbers. If anyone has tips on what to measure, which then becomes input for those config settings, that would be truly great. – JayMcTee Mar 14 '16 at 11:47
  • @JayMcTee Take a look here too - a fair few extra bits linode.com/docs/websites/nginx/… – Drifter104 Mar 14 '16 at 12:05
  • Thanks, very helpful. In New Relic Nginx plugin, I see the peak traffic is > 400 per 10 minutes. Avg user session is just over 3 minutes. So any given minute, there can be 120 people online. A popular page like the homepage takes 180 https requests. Over http/2 those are virtually simultaneously. So am I thinking right we should cater for 120 x 180 = 21,600 worker processes? I suppose, divided by the number of workers. – JayMcTee Mar 14 '16 at 12:19
  • @JayMcTee worker_processes = number of processors and max clients = worker_processes * worker_connections so almost. – Drifter104 Mar 14 '16 at 13:41
  • Thanks, you are right, I meant to say 21,600 worker connections. We have 16 worker processes / CPU cores so I was thinking 21,600 / 16 is 1,350 worker connections. Currently worker_rlimit_nofile and worker_connections is 1,024 which seems to get maxed out (and which seems wrong anyway, rlimit_nofile should be higher I believe especially since Nginx serves quite a few static files). Of course, keep_alive allows multiple requests through the same connection, so that muddies the waters a bit in terms of exact calculations. – JayMcTee Mar 14 '16 at 14:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.