1

I'm seeing average load times of load average: 12.41, 11.94, 11.59 on a linux-based machine serving a web application. It has 16 cores, so the load average isn't unmanageably high.

However, this web application is frequently timing out when I try to connect to it at the moment. What could be causing this? This is a bit of a curveball.

The CPU usage is hovering around ~50% for all CPUs (according to top). Values for wa are between 0.0 and 3.0. No swap memory is being used at all, and there's a ton of free mem available.

iostat shows an %iowait value of 0.51. Other stats are:

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda               4.88         1.02      2136.25   12365497 25895371840
sdb               0.00         0.00         0.00       9456          0
sdc               0.95         0.00       452.44       4781 5484405440

Writes/second are high - this is a write/heavy application. iotop shows writes coming from the pgbouncer process (a postgresql connection pooler), from async task queues and from nginx worker processes (probably writing to the access log). I don't see anything above 6% in the IO> column - and most rows have 0.00%. SWAPIN is 0.00% throughout.


In short, CPU utilization isn't through the roof, memory utilization isn't the problem, and there aren't signs of excessive I/O related waiting going on. Why would the web app be infinitely loading/timing out when I try to access it? Could it be issues in sysctl.conf or with my webserver? Need an expert opinion here.


The server in question is Ubuntu 14.04 LTS. Nginx is the webserver, used as a reverse proxy with Gunicorn (Django-based web application). The back-end is Postgresql 9.3, and Redis is in play as well. The DB resides in a separate VM.

0

If you are working with a large volume of TCP connections and are going through a reverse-proxy like nginx, you may be encountering TCP Port Exhaustion. In short, there are theoretically 65535 TCP ports. If you have one reverse proxy coming from IP 192.168.1.1 connecting to your web server on port 80 at 192.168.1.2:80, you can therefore make a theoretical maximum number of simultaneous connections through your reverse proxy of 65535 to port 80 on your web server. After that you run out of source ports (known as ephemeral ports) to use.

But it is a little more complicated than that: Linux by default is tuned to only use about 30000 (lower for older kernels/distros - as low as 1024) of these ports and even then it uses an algorithm that will randomly try to find a free source port for use. The closer you get to that 30000 mark, the more attempts the kernel makes to randomly select a free port and the longer it takes to find one. Try using netstat, grep and wc to count the number of connections you have and if you are nearing 30,000 this is likely the cause of your timeouts. You can review NGINX's suggestions for resolving this issue if this is the case.

  • 1
    Hmm, I actually didn't know how those ephemeral ports were being utilized. I'll give your suggestion a go. – Hassan Baig Sep 23 '17 at 9:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.