I am trying to run HAProxy (1.4.8) on a EC2 machine running Ubuntu 10.04. I need HAProxy to be able to handle many thousands of long-running persistent connections (websockets). With the current setup HAProxy gets killed at around 54,300 connections (roughly).

If I am running HAProxy in the foreground, the only output is "Killed". Am I right in thinking this is the Kernel killing the process? Is this because it is out of resources? Can I increase the resources?

The CPU and memory consumption are low with 50,000 connections, so I don't suspect either of these.

How can I prevent this from happening?

  • check /proc/sys/fs/file-max. on different systems can be anywhere from 50,000 to 500,000 – Javier Jan 12 '11 at 14:38
  • /proc/sys/fs/file-max = 174616 – Olly Jan 12 '11 at 14:56
  • I upped the limit to 350000 like this echo "350000" >> /proc/sys/fs/file-max and the problem still persisted. – Olly Jan 12 '11 at 15:03
  • Did you check the system logs /var/log/messages and /var/log/syslog? – Khaled Jan 12 '11 at 15:35
  • This is the output from /var/log/messages: gist.github.com/776339 Out of socket memory seems like it might be relevant. – Olly Jan 12 '11 at 15:56

Switching to a 64-bit instance solved the problem. I'm guessing we were running into the per-process memory limit on a 32-bit instance.

  • 1
    At 54000 concurrent connections, you should be careful about your TCP settings. If running with default settings (87kB read buffer, 16kB write buffer), you can end up eating 10 gigs of memory just for the sockets. It is very likely that in 32-bit, the system encountered memory allocation issues and decided to kill the most hungry user-space process, which "dmesg" should tell you. – Willy Tarreau Jan 19 '11 at 5:30
  • @WillyTarreau - could you point us to any information about how TCP settings should be adjusted for very high concurrency situations? (i.e. how to avoid such high memory usage). Thanks. – UpTheCreek Jan 14 '13 at 13:10
  • @WillyTarreau - ah sorry, I see there is some info in your post which ochoto linked to (serverfault.com/questions/106537/…) – UpTheCreek Jan 14 '13 at 13:13

You should check this answer from Wily Tarreau to the very same question:

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.