HAProxy is swapping (Major Faults) when Serving Web Sockets

Question

I've observed some swapping in our HAProxy instance that serves web sockets. The fault rate is low (0.01 major faults/s) currently. We use nbproc mode with one process for the http processing, and 3 other processes dedicated to SSL processing.

From Perf, I was able to grab the following fault samples from the http processing instance:

 Samples: 36  of event 'page-faults:u', Event count (approx.): 206
 28.64%  haproxy-t3  haproxy       [.] si_conn_wake_cb
 20.87%  haproxy-t3  haproxy       [.] si_conn_recv_cb
 13.11%  haproxy-t3  haproxy       [.] raw_sock_to_buf
 10.68%  haproxy-t3  haproxy       [.] stream_int_chk_snd_conn
  7.28%  haproxy-t3  haproxy       [.] conn_fd_handler
  4.37%  haproxy-t3  haproxy       [.] http_end_txn
  3.88%  haproxy-t3  haproxy       [.] stream_int_update_conn
  3.88%  haproxy-t3  haproxy       [.] process_session
  2.91%  haproxy-t3  haproxy       [.] eb_delete
  2.43%  haproxy-t3  haproxy       [.] stream_sock_read0
  1.94%  haproxy-t3  libc-2.12.so  [.] __memset_sse2

Since this maintains a fair ammount of conncurent connections, memory usage is fairly high (~16 GB for all instances (there are 4 total because we are running with nbproc).

Should I being try to prevent this faulting by setting swapiness to zero? I figure this could be healthy memory management but maybe haproxy should never really be swapping?

Reference Data:

Memory overhead on this machine:

[root@ny-lb06 ~]# free -m
             total       used       free     shared    buffers     cached
Mem:         64375      58876       5499          0         86      34472
-/+ buffers/cache:      24317      40058
Swap:         6015        267       5748

Version info:

HA-Proxy version 1.5.2 2014/07/12
Copyright 2000-2014 Willy Tarreau <[email protected]>

Build options :
  TARGET  = linux26
  CPU     = generic
  CC      = gcc
  CFLAGS  = -m64 -march=x86-64 -O2 -g -fno-strict-aliasing
  OPTIONS = USE_GETADDRINFO=1 USE_REGPARM=1 USE_OPENSSL=1 USE_STATIC_PCRE=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200

Encrypted password support via crypt(3): yes
Built without zlib support (USE_ZLIB not set)
Compression algorithms supported : identity
Built with OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013
Running on OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 7.8 2008-09-05
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with transparent proxy support using: IP_TRANSPARENT IP_FREEBIND

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Config Snippets:

global
    maxconn 300000
    tune.bufsize 16384
    nbproc 4

Memory of All HAProxy Instances (Note haproxy-t3 is our socket server instance and is the one that is swapping):

[root@ny-lb06 ~]# ps -A -o cmd,vsz,rss,pid
/opt/haproxy/haproxy-t3 -D  8424224 8299192 30343
/opt/haproxy/haproxy-t3 -D  2259988 2185768 30344
/opt/haproxy/haproxy-t3 -D  3079456 3013344 30345
/opt/haproxy/haproxy-t3 -D  2445524 2380072 30346
/opt/haproxy/haproxy-t4 -D   93332 27780 31606
/opt/haproxy/haproxy-t4 -D   61108  2988 31607
/opt/haproxy/haproxy-t4 -D   61232  3132 31608
/opt/haproxy/haproxy-t4 -D   61288  7464 31609
/opt/haproxy/haproxy-t2 -D   66572 14216 32497
/opt/haproxy/haproxy-t2 -D   63308 12052 32498
/opt/haproxy/haproxy-t2 -D   66400 15696 32499
/opt/haproxy/haproxy-t2 -D   64168 12592 32500
/opt/haproxy/haproxy-t20 -D  57400  5268 33284
/opt/haproxy/haproxy-t20 -D  59620  3864 33285
/opt/haproxy/haproxy-t20 -D  59640  6176 33286
/opt/haproxy/haproxy-t20 -D  59620  3928 33287
/opt/haproxy/haproxy-t1 -D  805556 750948 34693
/opt/haproxy/haproxy-t1 -D  189860 137264 34694
/opt/haproxy/haproxy-t1 -D  196988 144472 34696
/opt/haproxy/haproxy-t1 -D  187136 134524 34697
/opt/haproxy/haproxy-t5 -D   59464  7368 41065
/opt/haproxy/haproxy-t5 -D   59756  1772 41066
/opt/haproxy/haproxy-t5 -D   59984  2136 41067
/opt/haproxy/haproxy-t5 -D   59756  4240 41068

Answer 1

that must absolutely be prevented from happening! Fortunately you noticed it before it was too serious.

Please check your maxconn in the global section, check if you're using "tune.bufsize" in the global section (otherwise you can assume 16kB), and chec kthe number of processes.

The max amount of memory in use is about ((2*bufsize + 2kB) * maxconn * nbproc) for haproxy itself, plus a min of about (4 * 4kB * maxconn * nbproc) for the kernel-side sockets.

The problem with websocket is that connections can last long and stack together resulting in more stress than when doing HTTP where connections are very short. It is possible that your settings allow for too high a memory usage and that only WebSocket is capable of reaching these limits.

BTW, I'm seeing 34 GB of cache on this machine, so it is possible that it's real cache (in which case you shouldn't worry) or temporary data in /dev/shm. Also, could you please check the VSZ and RSS of your haproxy processes ?

Answer 2

After some digging, here is what I turned up:

The box definitely wasn't under memory contention of any sort. Nearly all of the cached data consisted of pages mapped to haproxy log files. As those files were constantly being written, and tended to be pretty beefy, they'd occupy a huge amount of cached pages.

When these disk pages for logs would get mapped, they'd end up swapping out really old anonymous pages from haproxy. All the really old pages happened to belong to our websockets haproxy, and they were likely buffer spaces from really old connections.

I've turned down swappiness so that it doesn't swap pages out quite as aggressively. This should result in those haproxy logs pages being discarded rather than haproxy anonymous pages being swapped out.