Testing SSL performance on Nginx

Question

I'm running nginx 1.0.0 on CentOS 6 x86_64 with stock OpenSSL. Below are results of openssl benchmarking.

sh# openssl speed aes-256-cbc
OpenSSL 1.0.0-fips 29 Mar 2010
built on: Sat Jun 25 04:58:15 BST 2011
options:bn(64,64) md2(int) rc4(1x,char) des(idx,cisc,16,int) aes(partial) blowfish(idx) 
compiler: gcc -fPIC -DOPENSSL_PIC -DZLIB -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN 
-DHAVE_DLFCN_H -DKRB5_MIT -m64 -DL_ENDIAN -DTERMIO -Wall -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 
-fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -Wa,--noexecstack 
-DMD32_REG_T=int -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM 
-DMD5_ASM -DAES_ASM -DWHIRLPOOL_ASM
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-256 cbc      51869.80k    54173.06k    54835.11k    54890.84k    55206.96k

AES-NI engine is enabled (I'm using Xeon E5620 @ 2.40GHz x 2):

sh# openssl engine -t
(aesni) Intel AES-NI engine
     [ available ]

And I also get same result using openssl speed -engine aesni aes-256-cbc

But when I use EVP:

sh# openssl speed -evp aes-256-cbc
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-256-cbc     403447.23k   420048.47k   424418.65k   425523.88k   426726.49k

So performance gain is significant. I found article about outmoded openssl assembly where Simon tested openssl without ASM for AES and numbers are:

[openssl-1.0.0d]# OPENSSL_CONF=apps/openssl.cnf util/opensslwrap.sh speed aes-256-cbc
OpenSSL 1.0.0d 8 Feb 2011
options:bn(64,64) rc4(1x,char) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx) 
compiler: gcc -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H 
-m64 -DL_ENDIAN -DTERMIO -O3 -Wall -DMD32_REG_T=int -DOPENSSL_IA32_SSE2 
-DOPENSSL_BN_ASM_MONT -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DWHIRLPOOL_ASM
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-256 cbc     107854.91k   111229.18k   112361.56k   112501.08k   112536.23k

I still haven't managed to build aesni engine on 1.0.0d to see the difference. And using Intel's IPP require skills for patching openssl.

So my questions are:

What is the proper way to test if nginx is using EVP for AES-256 or is it worth to compile nginx against openssl without asm for AES?

I know that every new connection will require RSA decryption to exchange the secret key, so thats the bottleneck that should be taken into account, but how much ssl_session_cache shared affects SSL session reusing and could tools like ab, siege or similar simulate real traffic ssl traffic?

Answer 1

What I have investigated may be helpful for you. There is an article from intel : https://software.intel.com/sites/default/files/open-ssl-performance-paper.pdf

They use apache to test and they said they did not tune any configuration of apache. I think for Nginx is same.

I use gdb to trace back and here is my result :

(gdb) bt
#0  0x00000000004dcd50 in aesni_init_key ()
#1  0x00000000004d8dff in EVP_CipherInit_ex ()
#2  0x0000000000494c5a in ssl3_send_newsession_ticket ()
#3  0x00000000004997e8 in ssl3_accept ()
#4  0x00000000004281af in ngx_ssl_handshake (c=0x7ffff7fad1c0) at src/event/ngx_event_openssl.c:996
#5  0x0000000000428571 in ngx_ssl_handshake_handler (ev=0x8c3770) at src/event/ngx_event_openssl.c:1144
#6  0x0000000000424467 in ngx_epoll_process_events (cycle=0x89b9d0, timer=<value optimized out>, flags=<value optimized out>)
    at src/event/modules/ngx_epoll_module.c:691
#7  0x000000000041bd43 in ngx_process_events_and_timers (cycle=0x89b9d0) at src/event/ngx_event.c:248
#8  0x0000000000421de8 in ngx_single_process_cycle (cycle=0x89b9d0) at src/os/unix/ngx_process_cycle.c:315
#9  0x000000000040519c in main (argc=<value optimized out>, argv=<value optimized out>) at src/core/nginx.c:404

EVP_CipherInit_ex uses ctx->cipher->init(ctx,key,iv,enc) to start aesni_init_key (). The details defined at openssl/crypto/evp/e_aes.c

 #define BLOCK_CIPHER_generic(nid,keylen,blocksize,ivlen,nmode,mode,MODE,flags) \
static const EVP_CIPHER aesni_##keylen##_##mode = { \
    nid##_##keylen##_##nmode,blocksize,keylen/8,ivlen, \
    flags|EVP_CIPH_##MODE##_MODE,   \
    aesni_init_key,         \
    aesni_##mode##_cipher,      \
    NULL,               \
    sizeof(EVP_AES_KEY),        \
    NULL,NULL,NULL,NULL }; \
static const EVP_CIPHER aes_##keylen##_##mode = { \
    nid##_##keylen##_##nmode,blocksize, \
    keylen/8,ivlen, \
    flags|EVP_CIPH_##MODE##_MODE,   \
    aes_init_key,           \
    aes_##mode##_cipher,        \
    NULL,               \
    sizeof(EVP_AES_KEY),        \
    NULL,NULL,NULL,NULL }; \
const EVP_CIPHER *EVP_aes_##keylen##_##mode(void) \
{ return AESNI_CAPABLE?&aesni_##keylen##_##mode:&aes_##keylen##_##mode; }

AESNI_CAPABLE determines which function enable, aes_init_key or aes_init_key. This is completed at compilation. Further detail you can find here .

If your openssl evp interface enabled AESNI, Nginx also uses that. So for your case, I think nginx is using AESNI by default.

Answer 2

I don't have any answer to your EVP question, but as far as SSL session caching goes, the answer is "it helps considerably". I've recently written a patch for nginx to implement SSL session caching across clusters using memcached, and I've found that there's considerable benefit to SSL session caching -- you're saving yourself several round trips (which is the important thing from the user's perspective, because they get their pages faster), as well as a moderate improvement in CPU usage (although it's rare that modern servers are CPU bound, so it's not usually an issue).

Testing that you're using SSL session caching is easy:

gnutls-cli -V -r HOSTNAME |grep 'Session ID'

But doing testing to see how much of an effect it has on "real traffic" is very difficult, simply because "real traffic" is so complex. Given how low risk the config change is, I'd recommend collecting some good statistics (based on whatever you want to improve), then turning it on in production and measuring again to see if your metrics improve.