0

I have an Angular website with static assets of around 1.5 mb and gzipped it is around 400 kb, I have nginx as my webserver & reverse proxy to the API server, when I test nginx with Apache benchmark tool, I find huge drop in performance if I test the https site compared to the http (https is 10 times slower) & the cpu utilization & memory is not high at all (cpu 30% memory is only 1 mb!!)

I have been searching for hours & tried all possible enhancements but none worked, as far as I have read https shall not be that much slower on modern web servers (http around 1500 req/sec & https is 46 req/sec for nginx), this is mostly from the Nginx https very high connect time but I have no clue how to solve this.

Can someone advise how to improve this?

(Also to my surprise, Apache performs much better in both cases but doesn't respond if I set concurrent connections to more than 200) & this is not nginx vs apache I am just stating my situation.

Important note:

I am not comparing the 2 web servers that is not the point of this site, but generally they have comparable performance so if https in nginx is 10 times slower than Apache I feel that something is wrong in my Nginx configuration & I want to fix it.

All test are on my windows machine i7 & 16 gb ram.

Nginx http only:

C:\Apache24\bin>ab -n 5000 -c 200 http://localhost:8100/abc/index.html?param=abc
This is ApacheBench, Version 2.3 <$Revision: 1826891 $>

Server Software:        nginx/1.15.4
Server Hostname:        localhost
Server Port:            8100

Document Path:          /abc/index.html?param=abc
Document Length:        1099 bytes

Concurrency Level:      200
Time taken for tests:   3.246 seconds
Complete requests:      5000
Failed requests:        0
Total transferred:      6665000 bytes
HTML transferred:       5495000 bytes
Requests per second:    1540.32 [#/sec] (mean)
Time per request:       129.843 [ms] (mean)
Time per request:       0.649 [ms] (mean, across all concurrent requests)
Transfer rate:          2005.12 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   1.3      0      16
Processing:    31   87  12.8     94     124
Waiting:        0   87  13.7     94     124
Total:         31   87  12.8     94     124

Percentage of the requests served within a certain time (ms)
  50%     94
  66%     94
  75%     94
  80%     94
  90%     99
  95%    109
  98%    109
  99%    113
 100%    124 (longest request)

Nginx https (with http2 enabled)

C:\Apache24\bin>abs -n 5000 -c 200 https://localhost:8200/abc/index.html?param=abc
This is ApacheBench, Version 2.3 <$Revision: 1826891 $>

Server Software:        nginx/1.15.4
Server Hostname:        localhost
Server Port:            8200
SSL/TLS Protocol:       TLSv1.2,ECDHE-RSA-AES256-GCM-SHA384,2048,256
TLS Server Name:        localhost

Document Path:          /abc/index.html?param=abc
Document Length:        1099 bytes

Concurrency Level:      200
Time taken for tests:   108.985 seconds
Complete requests:      5000
Failed requests:        0
Total transferred:      6780000 bytes
HTML transferred:       5495000 bytes
Requests per second:    45.88 [#/sec] (mean)
Time per request:       4359.386 [ms] (mean)
Time per request:       21.797 [ms] (mean, across all concurrent requests)
Transfer rate:          60.75 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       16 4201 506.8   4251    4755
Processing:     0   32  12.6     31      88
Waiting:        0   32  12.6     31      88
Total:         62 4232 506.9   4283    4800

Percentage of the requests served within a certain time (ms)
  50%   4283
  66%   4342
  75%   4413
  80%   4439
  90%   4484
  95%   4547
  98%   4694
  99%   4727
 100%   4800 (longest request)

Compared to Apache http (here CPU is around 90 to 100% utilized)

C:\Apache24\bin>ab -n 5000 -c 200 http://localhost:6200/abc/index.html?param=abc
Server Software:        Apache/2.4.33
Server Hostname:        localhost
Server Port:            6200

Document Path:          /abc/index.html?param=abc
Document Length:        1099 bytes

Concurrency Level:      200
Time taken for tests:   1.781 seconds
Complete requests:      5000
Failed requests:        0
Total transferred:      6810000 bytes
HTML transferred:       5495000 bytes
Requests per second:    2806.99 [#/sec] (mean)
Time per request:       71.251 [ms] (mean)
Time per request:       0.356 [ms] (mean, across all concurrent requests)
Transfer rate:          3733.51 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   1.6      0      16
Processing:    16   69  16.0     63     125
Waiting:        0   57  16.0     63     125
Total:         16   69  16.0     63     125

Percentage of the requests served within a certain time (ms)
  50%     63
  66%     78
  75%     78
  80%     78
  90%     94
  95%     94
  98%     94
  99%    109
 100%    125 (longest request)

And Apache https is as follows (http 1.1) & note that http 1.1 in nginx didn't improve its performance:

C:\Apache24\bin>abs -n 5000 -c 200 https://localhost:7200/abc/index.html?param=abc

This is ApacheBench, Version 2.3 <$Revision: 1826891 $>

Server Software:        Apache/2.4.33
Server Hostname:        localhost
Server Port:            7200
SSL/TLS Protocol:       TLSv1.2,ECDHE-RSA-AES256-GCM-SHA384,2048,256
TLS Server Name:        localhost

Document Path:          /abc/index.html?param=abc
Document Length:        1099 bytes

Concurrency Level:      200
Time taken for tests:   8.747 seconds
Complete requests:      5000
Failed requests:        0
Total transferred:      6810000 bytes
HTML transferred:       5495000 bytes
Requests per second:    571.60 [#/sec] (mean)
Time per request:       349.894 [ms] (mean)
Time per request:       1.749 [ms] (mean, across all concurrent requests)
Transfer rate:          760.27 [Kbytes/sec] received

Connection Times (ms)
            min  mean[+/-sd] median   max
Connect:        0  198  42.7    188     391
Processing:    62  145  39.1    140     385
Waiting:        0   76  28.3     78     250
Total:         62  343  63.0    331     615

Percentage of the requests served within a certain time (ms)
50%    331
66%    369
75%    380
80%    389
90%    422
95%    465
98%    500
99%    536
100%    615 (longest request)

My nginx configuration:

worker_processes auto;

events {
    worker_connections  1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;
    keepalive_timeout  65;

    server {
        listen       8100;
        server_name  localhost;

        location / {
            root   html;
            index  index.html index.htm;
        }

        #error_page  404              /404.html;

        # redirect server error pages to the static page /50x.html
        #
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }

    }

    server {
    listen       8200 ssl http2;
    server_name  localhost;

    ssl_certificate      C:/nginx-1.13.12/conf/server.crt;
    ssl_certificate_key  C:/nginx-1.13.12/conf/server.key;

        ssl_session_cache   shared:SSL:10m;
        ssl_session_timeout 10m;

    ssl_ciphers  HIGH:!aNULL:!MD5;
    ssl_prefer_server_ciphers  on;

        gzip on;
        gzip_comp_level 1;
        gzip_vary on;
        gzip_types
            text/css
            text/javascript
            text/xml
            text/plain
            text/x-component
            application/javascript
            application/json
            application/xml
            application/rss+xml
            font/truetype
            font/opentype
            application/vnd.ms-fontobject
            image/svg+xml;

        gzip_static on;

    location /ipo_reits/ {

        root   html;
        index  index.html index.htm;
        ## here we redirect to the homepage in case of nginx 404 
        try_files $uri $uri/ /ipo_reits/index.html;
        #    error_page 404 =301 /;
    }

        location /api/ {
            proxy_pass  https://localhost:7001/;
        }
    }
}

Update

I have tested on Linux as advised by kworr & nginx performance was much better in https

Nginx http: 17000 req/sec & https around 1200 req/sec, apache has a similar performance ratio between https 1150 req/sec & http 13000 req/sec

Why is that huge difference between http & https in both web servers ?

Http is around 12 times faster than https though many references are assuring difference shall be negligible?

  • This can be a cipher difference too. Can you check what ciphers apache uses when serving data? Or do attach apache config. PS: I don't think nginx-on-windows is production ready, it's not suppported by the team. To get meaningful results please retry same test on some unix host. Or please do check how connections are accepted and what you can tweak in listen clause. – kworr Oct 2 '18 at 15:03
  • @kworr Thanks for your help, actually the apache benchmark tool shows in the output shown above that nginx & apache are using the same cipher SSL/TLS Protocol: TLSv1.2,ECDHE-RSA-AES256-GCM-SHA384,2048,256 I didn't know nginx is not prod ready, I will try to redo the test on linux & update the post – Ahmed Elkoussy Oct 2 '18 at 15:51
  • @kworr thanks a lot for your help, performance on linux on same machine was way higher, I have posted an answer showing that to help anyone who will face the same, thanks – Ahmed Elkoussy Oct 2 '18 at 18:30
  • Since the core question is still valid, as http even in linux is 12 times faster, I updated the question & removed my answer, I hope someone can explain to me why http is that much faster than https & if that is the normal case – Ahmed Elkoussy Oct 2 '18 at 19:55
  • https means your data will be encrypted and decrypted, that's a lot of work added. – kworr Oct 2 '18 at 21:02
0

HTTPS will be considerably slower than HTTP for this simply test case.

HTTPS means two things:

  1. You need an initial set up to agree the encryption parameters (the TLS handshake).
  2. You need compute power to encrypt each request before sending and decrypt each request after receiving.

Now point 2 is negligible these days with the correct set up and for most traffic. Large volumes of traffic (e.g. if you were delivering mostly video like Netflix or YouTube) might notice a difference but most sites won’t really unless on really antiquated hardware or really bad set up.

Point one however is interesting. TLSv1.2, that you are using, requires 2 full round trips to establish the HTTPS connection on top of the 1.5 round trips required for a TCP connection and the 1 round trip for the actual HTTP request. TLSv1.3 reduces this to one round trip (or even zero if re-establishing a connection) and that’s one of the many reasons people are getting excited by it and it is starting to be come available in implementations now.

Therefore, for now, I would expect HTTPS to be almost twice as slow as HTTP at the very least - ignoring the encryption processing cost itself completely and assuming the download of the web page is small, as you are comparing 4.5 round trips to 2.5 round trips.

However the calculations do have a cost. HTTPS uses public key asymmetric encryption based on the TLS certificate provided by the site. This is needed because you don’t have a pre-agreed secret key you can use with every website in the world (especially those ones by yet created!) so you cant use the faster symmetric encryption. Asymmetric encryption is notoriously slow, so to help counteract that, this is only used to establish an symmetric key which is then used for the rest of the encryption. However it means the initial handshake again will be considerably slower because it is doing that expensive public key encryption and decryption.

I’ve never measure the difference using ab to be honest but I’m not completely surprised by it being 10-13 times slower for these reasons compared to using HTTP which just requires establishing a TCP connection and sending the page back.

So why the heck are any websites using HTTPS and how come most people say the performance impact is not that much?!?! Well that is because very few websites load just one page. A website is usually made of a myriad of resources (75 on average) and only the first one for each domain is hit by that handshake delay.

Therefore your ab test is not representative of the real word (unless you have a VERY simple website). Instead you should use wget -p or similar to download both the web page and the dependencies. Then time that and only then can you see what the difference is. I would still expect a difference but not as much as you have seen with your simple ab test.

However even that is not representative as downloading is only one part. If it takes 1 second to download your site and 15 seconds to chug through all the JavaScript, advertising tracking...etc. then even a doubling of that 1 second download time isn’t going to make a material difference. So to be a truly accurate test you should measure both downloading and rendering to get a true cost. Tools like webpagetest can help with this.

You should also ensure your HTTPS set up is optimised to use faster ciphers, ensure TLS resumption is enabled (so any subsequent connections don’t suffer as much of a performance penalty) and other settings.

Ultimately we are moving to an HTTPS world for good reasons and while there is a performance impact for that initial set up, there are also other performance benefits like Brotli compression and HTTP/2 which are only available over HTTPS (for both idealogical and practical reasons). These can leave HTTP in the dust when it comes to performance though often sites showcases this (like that one of mine) are often based on artificial tests that are not really representative of most websites so most websites won’t see quite that much performance improvement!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.