0

I'm tracing down a performance issue with SSL requests.

We run two webservers on two EC2 instances (us-east-2a/us-east-2b) with an ALB who also does SSL termination there, Route53 is in charge of the domain with a CNAME to the ALB's CNAME. Everything runs on a private VPC, with two private subnets, both subnets have a route table with Internet access through a NAT gateway. I'm using a VPN to reach the balancer/EC2 endpoints.

Hitting directly to the ALB using HTTP (no HTTP to HTTPS redirect),

% ab -n10 -c1 \
    -H "Host: service.internal.stg" \
    http://service.internal.stg/

This is ApacheBench, Version 2.3 <$Revision: 1843412 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking service.internal.stg (be patient).....done


Server Software:        Skipper
Server Hostname:        service.internal.stg
Server Port:            80

Document Path:          /
Document Length:        199 bytes

Concurrency Level:      1
Time taken for tests:   5.015 seconds
Complete requests:      10
Failed requests:        1
   (Connect: 0, Receive: 0, Length: 1, Exceptions: 0)
Non-2xx responses:      10
Total transferred:      4059 bytes
HTML transferred:       1989 bytes
Requests per second:    1.99 [#/sec] (mean)
Time per request:       501.536 [ms] (mean)
Time per request:       501.536 [ms] (mean, across all concurrent requests)
Transfer rate:          0.79 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:      203  251  51.0    243     315
Processing:   216  251  43.5    221     309
Waiting:      216  250  43.5    221     309
Total:        420  501  77.9    520     617

Percentage of the requests served within a certain time (ms)
  50%    520
  66%    536
  75%    550
  80%    612
  90%    617
  95%    617
  98%    617
  99%    617
 100%    617 (longest request)

Hitting directly to the ALB using HTTPS,

% ab -n10 -c1 \
    -H "Host: service.internal.stg" \
    http://service.internal.stg/

This is ApacheBench, Version 2.3 <$Revision: 1843412 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking service.internal.stg (be patient).....done


Server Software:        Skipper
Server Hostname:        service.internal.stg
Server Port:            443
SSL/TLS Protocol:       TLSv1.2,ECDHE-RSA-AES128-GCM-SHA256,2048,128
Server Temp Key:        ECDH P-256 256 bits
TLS Server Name:        service.internal.stg

Document Path:          /
Document Length:        199 bytes

Concurrency Level:      1
Time taken for tests:   9.822 seconds
Complete requests:      10
Failed requests:        0
Non-2xx responses:      10
Total transferred:      4060 bytes
HTML transferred:       1990 bytes
Requests per second:    1.02 [#/sec] (mean)
Time per request:       982.242 [ms] (mean)
Time per request:       982.242 [ms] (mean, across all concurrent requests)
Transfer rate:          0.40 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:      633  737 100.4    792     883
Processing:   220  245  31.5    231     303
Waiting:      220  245  31.5    231     303
Total:        858  982 105.1   1039    1114

Percentage of the requests served within a certain time (ms)
  50%   1039
  66%   1041
  75%   1061
  80%   1108
  90%   1114
  95%   1114
  98%   1114
  99%   1114
 100%   1114 (longest request)

I got waaaaay higher connection times. But, running ab with HTTP Keepalive (-k), I only one slow request (~900ms) but on mean time we're pretty good reaching ~320ms.

% ab -n10 -c1 \
    -H "Host: service.internal.stg" \
    http://service.internal.stg/

This is ApacheBench, Version 2.3 <$Revision: 1843412 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking service.internal.stg (be patient).....done


Server Software:        Skipper
Server Hostname:        service.internal.stg
Server Port:            443
SSL/TLS Protocol:       TLSv1.2,ECDHE-RSA-AES128-GCM-SHA256,2048,128
Server Temp Key:        ECDH P-256 256 bits
TLS Server Name:        service.internal.stg

Document Path:          /
Document Length:        199 bytes

Concurrency Level:      1
Time taken for tests:   3.242 seconds
Complete requests:      10
Failed requests:        1
   (Connect: 0, Receive: 0, Length: 1, Exceptions: 0)
Non-2xx responses:      10
Keep-Alive requests:    10
Total transferred:      4109 bytes
HTML transferred:       1989 bytes
Requests per second:    3.08 [#/sec] (mean)
Time per request:       324.238 [ms] (mean)
Time per request:       324.238 [ms] (mean, across all concurrent requests)
Transfer rate:          1.24 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0   92 292.2      0     924
Processing:   217  232  22.7    223     279
Waiting:      217  232  22.6    223     279
Total:        217  324 289.5    224    1146

Percentage of the requests served within a certain time (ms)
  50%    224
  66%    227
  75%    269
  80%    279
  90%   1146
  95%   1146
  98%   1146
  99%   1146
 100%   1146 (longest request)

I'm in doubt with the SSL termination performance on the ALB then, but I'm not sure how to handle/work on this.

Additional information: - Ping from my location to the EC2 instance

% ping 10.1.1.95 -c 10                                                                                                                              ~
PING 10.1.1.95 (10.1.1.95): 56 data bytes
64 bytes from 10.1.1.95: icmp_seq=0 ttl=61 time=203.177 ms
64 bytes from 10.1.1.95: icmp_seq=1 ttl=61 time=202.369 ms
64 bytes from 10.1.1.95: icmp_seq=2 ttl=61 time=317.346 ms
64 bytes from 10.1.1.95: icmp_seq=3 ttl=61 time=232.651 ms
64 bytes from 10.1.1.95: icmp_seq=4 ttl=61 time=252.859 ms
64 bytes from 10.1.1.95: icmp_seq=5 ttl=61 time=271.837 ms
64 bytes from 10.1.1.95: icmp_seq=6 ttl=61 time=204.135 ms
64 bytes from 10.1.1.95: icmp_seq=7 ttl=61 time=208.154 ms
64 bytes from 10.1.1.95: icmp_seq=8 ttl=61 time=201.772 ms
64 bytes from 10.1.1.95: icmp_seq=9 ttl=61 time=208.608 ms

--- 10.1.1.95 ping statistics ---
10 packets transmitted, 10 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 201.772/230.291/317.346/37.138 ms
  • AB running from an EC2 instance in the same VPC
ubuntu@ip-10-1-11-72:~$ ab -n10 -c1 \
    -H "Host: service.internal.stg" \
    http://service.internal.stg/
This is ApacheBench, Version 2.3 <$Revision: 1807734 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking service.internal.stg (be patient).....done


Server Software:        Skipper
Server Hostname:        service.internal.stg
Server Port:            443
SSL/TLS Protocol:       TLSv1.2,ECDHE-RSA-AES128-GCM-SHA256,2048,128
TLS Server Name:        service.internal.stg

Document Path:          /
Document Length:        199 bytes

Concurrency Level:      1
Time taken for tests:   0.164 seconds
Complete requests:      10
Failed requests:        2
   (Connect: 0, Receive: 0, Length: 2, Exceptions: 0)
Non-2xx responses:      10
Total transferred:      4058 bytes
HTML transferred:       1988 bytes
Requests per second:    61.11 [#/sec] (mean)
Time per request:       16.363 [ms] (mean)
Time per request:       16.363 [ms] (mean, across all concurrent requests)
Transfer rate:          24.22 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        4    6   2.6      5      11
Processing:     8   11   2.1     11      15
Waiting:        8   11   2.1     11      15
Total:         12   16   4.0     15      24

Percentage of the requests served within a certain time (ms)
  50%     15
  66%     16
  75%     20
  80%     21
  90%     24
  95%     24
  98%     24
  99%     24
 100%     24 (longest request)
  • AB running from an EC2 instance in the same VPC, hitting the webserver.
ubuntu@ip-10-1-11-72:~$ ab -n10 -c1 -k \
>     -H "Host: service.internal.stg" \
>     http://10.1.1.95:9999/
This is ApacheBench, Version 2.3 <$Revision: 1807734 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 10.1.1.95 (be patient).....done


Server Software:        Skipper
Server Hostname:        10.1.1.95
Server Port:            9999

Document Path:          /
Document Length:        199 bytes

Concurrency Level:      1
Time taken for tests:   0.075 seconds
Complete requests:      10
Failed requests:        0
Non-2xx responses:      10
Keep-Alive requests:    10
Total transferred:      4110 bytes
HTML transferred:       1990 bytes
Requests per second:    133.79 [#/sec] (mean)
Time per request:       7.475 [ms] (mean)
Time per request:       7.475 [ms] (mean, across all concurrent requests)
Transfer rate:          53.70 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       0
Processing:     6    7   1.4      7      11
Waiting:        6    7   1.4      7      11
Total:          6    7   1.4      7      11

Percentage of the requests served within a certain time (ms)
  50%      7
  66%      8
  75%      8
  80%      9
  90%     11
  95%     11
  98%     11
  99%     11
 100%     11 (longest request)
ubuntu@ip-10-1-11-72:~$
Improve this question
2
  • Can you please edit your question to add the following 1) The ping time from your testing location to either ALB or one of your EC2 instances both over the VPN and if you can without the VPN 2) run an EC2 instance in the same VPC as the application servers and use ab to test for http / https to both the ALB and to one of the servers directly. Basically I want to understand latency and application performance, as latency increases https session establishment time.
    – Tim
    Jan 13, 2020 at 21:16
  • Thanks Tim. I've added the requested information. I'm not able to ping from outside the VPN
    – Nico
    Jan 14, 2020 at 13:54

1 Answer 1

Reset to default
1

Connection establishment requires few requests from the client to the server - depending on the version of TLS it's between 1 and 4, from memory.

Your latency to the server is 200 - 320ms, and is highly variable. The high latency is why SSL session establishment is slow from your location, and also explains why it's so much faster when run locally.

Solutions could include:

  • Locating the server closer to you or your users, or running multiple servers with geolocaion
  • Use CloudFront to do TLS termination / offload at the edge. It's probably not a great solution to do https termination at the edge, but perhaps CloudFront or a CDN can make this more efficient using a more optimised network.
  • Force newer versions of TLS, which are more efficient.
Improve this answer
1
  • 1
    Thanks. I ended up with a peering connection at my closest datacenter to do SSL offload there, which highly improved connection times by 3x
    – Nico
    Jan 14, 2020 at 19:17

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .