Testing SSL performance on Nginx

Question

I'm running nginx 1.0.0 on CentOS 6 x86_64 with stock OpenSSL. Below are results of openssl benchmarking.

sh# openssl speed aes-256-cbc
OpenSSL 1.0.0-fips 29 Mar 2010
built on: Sat Jun 25 04:58:15 BST 2011
options:bn(64,64) md2(int) rc4(1x,char) des(idx,cisc,16,int) aes(partial) blowfish(idx) 
compiler: gcc -fPIC -DOPENSSL_PIC -DZLIB -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN 
-DHAVE_DLFCN_H -DKRB5_MIT -m64 -DL_ENDIAN -DTERMIO -Wall -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 
-fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -Wa,--noexecstack 
-DMD32_REG_T=int -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM 
-DMD5_ASM -DAES_ASM -DWHIRLPOOL_ASM
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-256 cbc      51869.80k    54173.06k    54835.11k    54890.84k    55206.96k

AES-NI engine is enabled (I'm using Xeon E5620 @ 2.40GHz x 2):

sh# openssl engine -t
(aesni) Intel AES-NI engine
     [ available ]

And I also get same result using openssl speed -engine aesni aes-256-cbc

But when I use EVP:

sh# openssl speed -evp aes-256-cbc
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-256-cbc     403447.23k   420048.47k   424418.65k   425523.88k   426726.49k

So performance gain is significant. I found article about outmoded openssl assembly where Simon tested openssl without ASM for AES and numbers are:

[openssl-1.0.0d]# OPENSSL_CONF=apps/openssl.cnf util/opensslwrap.sh speed aes-256-cbc
OpenSSL 1.0.0d 8 Feb 2011
options:bn(64,64) rc4(1x,char) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx) 
compiler: gcc -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H 
-m64 -DL_ENDIAN -DTERMIO -O3 -Wall -DMD32_REG_T=int -DOPENSSL_IA32_SSE2 
-DOPENSSL_BN_ASM_MONT -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DWHIRLPOOL_ASM
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-256 cbc     107854.91k   111229.18k   112361.56k   112501.08k   112536.23k

I still haven't managed to build aesni engine on 1.0.0d to see the difference. And using Intel's IPP require skills for patching openssl.

So my questions are:

What is the proper way to test if nginx is using EVP for AES-256 or is it worth to compile nginx against openssl without asm for AES?

I know that every new connection will require RSA decryption to exchange the secret key, so thats the bottleneck that should be taken into account, but how much *ssl_session_cache shared* affects SSL session reusing and could tools like ab, siege or similar simulate real traffic ssl traffic?

Answer 1

I don't have any answer to your EVP question, but as far as SSL session caching goes, the answer is "it helps considerably". I've recently written a patch for nginx to implement SSL session caching across clusters using memcached, and I've found that there's considerable benefit to SSL session caching -- you're saving yourself several round trips (which is the important thing from the user's perspective, because they get their pages faster), as well as a moderate improvement in CPU usage (although it's rare that modern servers are CPU bound, so it's not usually an issue).

Testing that you're using SSL session caching is easy:

gnutls-cli -V -r HOSTNAME |grep 'Session ID'

But doing testing to see how much of an effect it has on "real traffic" is very difficult, simply because "real traffic" is so complex. Given how low risk the config change is, I'd recommend collecting some good statistics (based on whatever you want to improve), then turning it on in production and measuring again to see if your metrics improve.