haproxy-varnish can't reach 6k requests

Question

Issue

I can't reach 6k requests/seconds.
I get a lot of timeouts.
My response time goes as high as 26 seconds.

Intro

I am setting up a server that will host a 100MB static-website.
The thing is that I will have to handle about 8000 constant requests for 5 days in a row.

I made the following setup :

HAProxy -> Varnish -> Nginx -> Staticfiles

HAProxy handle the connections on port 80 (soon on port 443 as well), transferts the requests to Varnish that will serve the files from the cache. I have set Nginx to expires 7d;. So Narnish will request the static file to Nginx every 7 days.

• I use the highest compression level on Nginx so Varnish store highly compressed static files gzip_comp_level 9;.
• I have set the ttl of the static files to 7 days in Nginx expires 7d;.
• I have set a high number of thread (At least, I think) in Varnish thread_pools=8 thread_pool_max=4000.
• I have set a not to big but not to small memory for Varnish (static files are not bigger 100MB altogether) malloc,512m.
• I have set a max nuimber of maxconn in HAProxy maxconn 65000.
• I tried to play with the sysctl configs, but I'm not sure it changed anything, that's why I think my problem come from my configurations.
• I believe HAProxy with maxconn 65000 doesn't throttle. I think Varnish throttle my requests, but I'm not sure how to confirm it.

my server is set like so:
Intel(R) Xeon(R) CPU E3-1245 V2 @ 3.40GHz
Number : 8
Cache : 8192 KB
Speed : 1764 MHz
RAM 2 x 8192 MB

Configuration files

Nginx

user www-data;
worker_processes 4;
pid /run/nginx.pid;

events {
  worker_connections 768;
}

http {
  sendfile on;
  tcp_nopush on;
  tcp_nodelay on;
  keepalive_timeout 65;
  types_hash_max_size 2048;
  include /etc/nginx/mime.types;
  default_type application/octet-stream;
  ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
  ssl_prefer_server_ciphers on;

  access_log /var/log/nginx/access.log;
  error_log /var/log/nginx/error.log;

  gzip on;
  gzip_disable "msie6";
  gzip_vary on;
  gzip_comp_level 9;
  gzip_buffers 16 16k;
  gzip_http_version 1.1;
  gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;


server {
  listen 127.0.0.1:82 default_server;
  root /var/www/html;

  index index.html index.htm index.nginx-debian.html;
  location / {
    expires 7d;
    add_header Cache-Control "public";
    # First attempt to serve request as file, then
    # as directory, then fall back to displaying a 404.
    try_files $uri $uri/ =404;
  }
}

}

Varnish

//DEAMON
DAEMON_OPTS="-a localhost:6081 \
             -T localhost:6082 \
             -f /etc/varnish/default.vcl \
             -S /etc/varnish/secret \
             -p thread_pools=8 \
             -p thread_pool_min=100 \
             -p thread_pool_max=4000 \
             -s malloc,512m"
//default.vcl
# new 4.0 format.
vcl 4.0;

# Default backend definition. Set this to point to your content server.
backend default {
  .host = "127.0.0.1";
  .port = "82";
}

HAProxy

global
  log /dev/log  local0
  log /dev/log  local1 notice
  chroot /var/lib/haproxy
  stats socket /run/haproxy/admin.sock mode 660 level admin
  stats timeout 30s
  user haproxy
  group haproxy
  daemon
  ca-base /etc/ssl/certs
  crt-base /etc/ssl/private
  ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
  ssl-default-bind-options no-sslv3
  maxconn 65000

defaults
  log global
  mode  http
  option  httplog
  option  dontlognull
  timeout connect 5000
  timeout client  50000
  timeout server  50000
  errorfile 400 /etc/haproxy/errors/400.http
  errorfile 403 /etc/haproxy/errors/403.http
  errorfile 408 /etc/haproxy/errors/408.http
  errorfile 500 /etc/haproxy/errors/500.http
  errorfile 502 /etc/haproxy/errors/502.http
  errorfile 503 /etc/haproxy/errors/503.http
  errorfile 504 /etc/haproxy/errors/504.http

userlist users
  group admin
  user username insecure-password password groups admin

frontend static_https
  bind *:80
  mode http
  acl aclok http_auth_group(users) admin
  #http-request auth realm admin if !aclok
  default_backend static_varnish

backend static_varnish
  mode http
  option forwardfor
  http-request set-header X-Forwarded-Port %[dst_port]
  http-request add-header X-Forwarded-Proto https if { ssl_fc }
  option httpchk HEAD / HTTP/1.1\r\nHost:localhost
  server varnish 127.0.0.1:6081 check

sysctl.conf

net.ipv4.tcp_max_syn_backlog = 10240
net.ipv4.tcp_fin_timeout = 30
net.ipv4.ip_local_port_range = 2048 61000
net.core.netdev_max_backlog = 40000
net.ipv4.tcp_max_tw_buckets = 400000
net.ipv4.tcp_max_orphans = 60000
net.core.somaxconn = 40000

EDIT

loader.io config load test and result

Answers

@fox

Why HAProxy in front of Varnish?

My client absolutely want https to appear in the browser. So I tought I would delegate the job of handling the ssl certificate to HAProxy.

What are the loads during test?

I haven't set a monitoring tool yet, but from the last test I ran, by looking at htop I can tell:
- proc: 25% avg.
- ram: 1070MB avg.

How are you testing?

I am using loader.io, it creates 10000 clients, and make them request for 1 minute. you can see the full test here: http://ldr.io/1eLKrrT

Using keep-alives?

I'm not sure how loader.io does.

What hardware?

I can't tell you more than what I wrote above, unless there is a way using some shell commands ?

Does the test pass Accept-Encoding: gzip?

I added it in loader.io after your comment, it didn't change anything.

What is the cache hit rate?

This is a very good question, but I don't know where to see that?

Answer 1

I will start this answer as a kind of open-answer. As I can not answer it all at once and there will be follow-up questions.

First thing is - you are kind of overcomplicating things. Serving static content through nginx, varnish and haproxy will probably cause a lot of unnecessary overhead (more TCP connections, more content switches, more memory used, etc.).

You may actually get better performance using directly nginx. As all of your content will fit into memory, nginx will be serving those files from memory anyway (thanks to OS cache). And using open_file_cache you even get around the filesystem overhead. Varnish could probably give you a slightly better performance for plain HTTP.

In your case, I would start with the KISS method - plain nginx, tune it to best performance. Then add SSL and tune that to best performance (keep-alives, ssl session caching). If you feel, that your performance with nginx alone is not good enough, then try figuring, what your real bottleneck is and how that can be mitigated (using what software).

And the benchmarking. I'd start with something local, like JMeter, or httperf, or at least Apache Bench. That helps you rule out all the things internet. Once you have figured out how your setup reacts to different kinds of tuning, then use something like loader.io to finetune the real world performance.

Some reasons that come to mind for the performance you are seeing now:

• you add latency to every connection, by putting afront of varnish (try hitting varnish directly)
• your varnish cache hit rate is not as good as you expect, with static content it should be 100% (use varnishstat to find out)
• you are starving some resource, open file descriptors, etc.
• if you are serving 7000 rps, but getting hit by 10000 rps, there are 3000 unserved requests. That's what causes the timeouts.
• logging can actually cause hiccups too. It should not be problem for Varnish (it's using shm and separate process), but I have no information about how logging is done in HAProxy and nginx
• we'll come up with more I think ...