Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I am using NGINX as a reverse web proxy to an upstream IIS webserver. I am proxy_pass'ing to an https binding on the IIS server so I understand SSL is being encrypted/decrypted twice- but it shouldn't be as slow as it is.

NGINX version 1.4.4

OpenSSL version 1.0.1f

Things I've tried:

  • Tweaking SSL cipher list
  • Recompiling NGINX with debugging on and & upgrading to latest OpenSSL
  • Inspecting access/error/debug logging output
  • Playing w/ various NGINX directives

Using ApacheBench for testing:

Requests per second:

  • HTTP (through nginx) = 3312 RPS
  • HTTPS (through nginx) = 273 RPS <-- ???
  • HTTP (direct to backend) = 4237 RPS
  • HTTPS (direct to backend) = 1349 RPS

Example ApacheBench output:

abs -c 100 -n 1000 h ttps://nginxtest1.mydomain.com/
Benchmarking nginxtest1.mydomain.com (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests
Server Software:        nginx/1.4.3
Server Hostname:        nginxtest1.mydomain.com
Server Port:            443
SSL/TLS Protocol:       TLSv1,ECDHE-RSA-AES256-SHA,2048,256
Document Path:          /
Document Length:        659 bytes
Concurrency Level:      100
Time taken for tests:   3.493 seconds
Complete requests:      1000
Failed requests:        0
Total transferred:      917000 bytes
HTML transferred:       659000 bytes
Requests per second:    286.27 [#/sec] (mean) <-----?????
Time per request:       349.320 [ms] (mean)
Time per request:       3.493 [ms] (mean, across all concurrent requests)
Transfer rate:          256.36 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       39  149  48.4    148     340
Processing:    47  180 179.2    163    3156
Waiting:       25  133 182.8    107    3130
Total:        145  330 181.7    320    3213

Percentage of the requests served within a certain time (ms)
  50%    320
  66%    331
  75%    339
  80%    355
  90%    431
  95%    502
  98%    529
  99%    533
 100%   3213 (longest request)

nginx.conf:

worker_processes  auto;

events {

 worker_connections  1024;
 debug_connection 192.168.2.98;

}

http {

   include       mime.types;
    default_type  application/octet-stream;

   sendfile        on;

   keepalive_timeout  65;
   lingering_time 240;
   client_max_body_size 100m;

   ssl_session_cache       shared:SSL:10m;


   include /usr/local/nginx/conf/srvnj04.conf;

   gzip on;
   gzip_disable "msie6";
   gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;

}

srvnj04.conf:

server {

   listen       80;
   server_name nginxtest1.mydomain.com;

     log_format custom4 '$remote_addr '
                        'Conn: $connection '                 #connection serial number
                        'Conn reqs: $connection_requests'    #the current number of requests made through a connection
                        '$status '
                        '$http_referer '
                        #'$body_bytes_sent '
                        '$request '
                        #'"$http_user_agent" '
                        'Processing: $request_time '
                        'Response: $upstream_response_time ';
                        #'$bytes_sent '
                        #'$request_length';


   access_log /var/log/nginx/srvnj04_access.log custom4;
   error_log /var/log/nginx/srvnj04_error.log debug;

    ssl                  off;

    location / {

            #empty_gif;
            proxy_pass http://nginxtest2.appsrv008.mydomain.com;

    }
}

upstream https_backend {

    server 192.168.2.4:444;

    #keepalive 32;
    keepalive 128;
}


server {

listen       443 ssl so_keepalive=0h:5m:0;
server_name  nginxtest1.mydomain.com;
keepalive_timeout 54;
keepalive_requests 128;

error_log /var/log/nginx/nginx_debug_srvnj04.log debug;

ssl                  on;

ssl_certificate ssl/star_mydomain_net_sol_CA_srvnj04b.cer;
ssl_certificate_key ssl/star_mydomain_net_sol.key;
ssl_dhparam ssl/dhparam.pem;

ssl_protocols  SSLv3 TLSv1.2 TLSv1 TLSv1.1;

ssl_ciphers RC4:HIGH:!aNULL:!MD5:!kEDH;

ssl_prefer_server_ciphers   on;

ssl_session_cache shared:SSL:10m;
ssl_session_timeout 10m;

location / {

#proxy_pass https://nginxtest1.mydomain.com;
proxy_pass https://https_backend;    

}

}

I have been looking into this for about a week now- any insight as to what might be wrong would be greatly appreciated.

As Requested, AB output:

HTTPS

# abs -c 100 -n 1000 #https://nginxtest1.mydomain.com/
This is ApacheBench, Version 2.3 <$Revision: 1528965 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking nginxtest1.mydomain.com (be patient)
Completed 100 requests
Completed 200 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests


Server Software:        nginx/1.4.3
Server Hostname:        nginxtest1.mydomain.com
Server Port:            443
SSL/TLS Protocol:       TLSv1,ECDHE-RSA-AES256-SHA,2048,256

Document Path:          /
Document Length:        334 bytes

Concurrency Level:      100
Time taken for tests:   3.458 seconds
Complete requests:      1000
Failed requests:        0
Non-2xx responses:      1000
Total transferred:      503000 bytes
HTML transferred:       334000 bytes
Requests per second:    289.17 [#/sec] (mean)
Time per request:       345.820 [ms] (mean)
Time per request:       3.458 [ms] (mean, across all concurrent requests)
Transfer rate:          142.04 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       54  186  73.7    174     509
Processing:    25  143  35.8    143     227
Waiting:       22   79  25.7     80     164
Total:         91  329  80.5    316     663

Percentage of the requests served within a certain time (ms)
  50%    316
  66%    318
  75%    323
  80%    334
  90%    404
  95%    542
  98%    568
  99%    595
 100%    663 (longest request)

HTTP

# abs -c 100 -n 1000 http://nginxtest1.mydomain.com/
This is ApacheBench, Version 2.3 <$Revision: 1528965 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking nginxtest1.mydomain.com (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests


Server Software:        nginx/1.4.3
Server Hostname:        nginxtest1.mydomain.com
Server Port:            80

Document Path:          /
Document Length:        659 bytes

Concurrency Level:      100
Time taken for tests:   0.772 seconds
Complete requests:      1000
Failed requests:        0
Total transferred:      917000 bytes
HTML transferred:       659000 bytes
Requests per second:    1295.26 [#/sec] (mean)
Time per request:       77.204 [ms] (mean)
Time per request:       0.772 [ms] (mean, across all concurrent requests)
Transfer rate:          1159.92 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.5      0       2
Processing:    33   68  27.7     53     296
Waiting:       12   65  28.4     53     145
Total:         34   69  27.7     54     297

Percentage of the requests served within a certain time (ms)
  50%     54
  66%     86
  75%     94
  80%     97
  90%    103
  95%    108
  98%    113
  99%    122
 100%    297 (longest request)
share|improve this question
    
Can you provide the output of the ab run against HTTP for comparison? And maybe see how the time per request changes when you increase and decrease the number of concurrent requests? –  Shane Madden Feb 11 '14 at 1:02
    
I have provided the AB output for both HTTPS and HTTP. 30 concurrent requests seems to be the sweet spot, which bring the RPS to ~290. –  arthur Feb 12 '14 at 2:57
    
Any difference with ssl_session_cache disabled? Are you maybe running out of entropy - watch /proc/sys/kernel/random/entropy_avail while ab is running? –  Shane Madden Feb 12 '14 at 3:24

1 Answer 1

up vote 0 down vote accepted

I understand SSL is being encrypted/decrypted twice- but it shouldn't be as slow as it is.

Why not? Please note, that it's not only double encryption/decryption, it also adds one more TCP+TLS handshake per request. To minimize handshake overhead you should use keepalive connections: http://nginx.org/r/keepalive

But in general what you're doing is a bad practise, you should use IPSec tunnel.

share|improve this answer
    
Thanks for the reply- would it really be adding one more TCP+TLS handshake per request with both 'ssl_session_cache'? I have tried enabling keepalive to upstream (ex: keepalive 32) - did not make a difference. I also have keepalive enabled in main http section. IPSec is certainly an option- but I'd like to understand why NGINX is performing so poorly in this scenario. –  arthur Feb 12 '14 at 3:02
    
All ssl_* directives have nothing to do with https connections to upstream. HTTPS support in proxy module is very limited. Also please read keepalive documentation carefully, just keepalive 32 in upstream block - isn't enough to enable HTTP/1.1 keeplive (not to mention that they barely help in bench with ab -c 100). –  VBart Feb 12 '14 at 15:05
    
please see my updated server.conf. Is this not adequate to enable keepalive to upstream? –  arthur Feb 12 '14 at 19:32
    
It is not adequate, instead it have so_keepalive in listen that is about TCP_KEEPALIVE and have nothing to do with HTTP keepalive, and your value for so_keepalive looks very strange. Just remove it. As I've already said you should read documentation on the link, it has full example: nginx.org/r/keepalive (For HTTP, ...) –  VBart Feb 12 '14 at 22:23
    
keepalive_timeout and keepalive_requests also are irrelevant to upstream HTTP keepalive. The fact that all these directives have "keepalive" word in name doesn't mean that they do right thing to you. And your lingering_time 240 value just looks wrong, I advise do not touch directives that you do not fully understand. –  VBart Feb 12 '14 at 22:29

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.