Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have an Amazon Web Services setup with an Apache instance behind Nginx with Nginx handling SSL and serving everything but the .php pages. In my ApacheBench tests I'm seeing this for my most expensive API call (which cache via Memcached):

100 concurrent calls to API call (http): 115ms (median) 260ms (max)
100 concurrent calls to API call (https): 6.1s (median) 11.9s (max)

I've done a bit of research, disabled the most expensive SSL ciphers and enabled SSL caching (I know it doesn't help in this particular test.) Can you tell me why my SSL is taking so long? I've set up a massive EC2 server with 8CPUs and even applying consistent load to it only brings it up to 50% total CPU. I have 8 Nginx workers set and a bunch of Apache. Currently this whole setup is on one EC2 box but I plan to split it up and load balance it. There have been a few questions on this topic but none of those answers (disable expensive ciphers, cache ssl, seem to do anything.) Sample results below:

$ ab -k -n 100 -c 100 https://URL 
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking URL.com (be patient).....done


Server Software:        nginx/1.0.15
Server Hostname:        URL.com
Server Port:            443
SSL/TLS Protocol:       TLSv1/SSLv3,AES256-SHA,2048,256

Document Path:          /PATH
Document Length:        73142 bytes

Concurrency Level:      100
Time taken for tests:   12.204 seconds
Complete requests:      100
Failed requests:        0
Write errors:           0
Keep-Alive requests:    0
Total transferred:      7351097 bytes
HTML transferred:       7314200 bytes
Requests per second:    8.19 [#/sec] (mean)
Time per request:       12203.589 [ms] (mean)
Time per request:       122.036 [ms] (mean, across all concurrent requests)
Transfer rate:          588.25 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       65  168  64.1    162     268
Processing:   385 6096 3438.6   6199   11928
Waiting:      379 6091 3438.5   6194   11923
Total:        449 6264 3476.4   6323   12196

Percentage of the requests served within a certain time (ms)
  50%   6323
  66%   8244
  75%   9321
  80%   9919
  90%  11119
  95%  11720
  98%  12076
  99%  12196
 100%  12196 (longest request)
share|improve this question
1  
How's the entropy looking? cat /proc/sys/kernel/random/entropy_avail –  Shane Madden Sep 21 '12 at 21:35
1  
Is that info gathered during the load testing, or while idle? You'll want to check it during the load test - sorry that I didn't mention that initially! –  Shane Madden Sep 22 '12 at 7:01
1  
Yeah, it ought to be getting closer to 0 if entropy were the problem. Not sure if this will help, but let's try a less expensive algorithm - try putting RC4+RSA first in your SSL cipher list, so that it will be preferred? –  Shane Madden Sep 22 '12 at 15:22
1  
Interesting.. let's rule out anything other than "slow SSL" - create a static document of around the same size of the response to your API calls to benchmark against? –  Shane Madden Sep 22 '12 at 18:50
1  
Nice, good find! I'll have to take you up on that next time I'm out there! –  Shane Madden Sep 23 '12 at 0:57
show 6 more comments

1 Answer

the issue ended up being that I had set Nginx rate limiting early on and it was affecting my load tests. Foolish I know. One interesting thing during rate limiting is that Nginx doesn't immediately get 503 errors past the burst threshold but just delays the responses. Only when there are too many requests in queue will it send 503's.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.