I'm seeing average load times of load average: 12.41, 11.94, 11.59
on a linux-based machine serving a web application. It has 16 cores, so the load average isn't unmanageably high.
However, this web application is frequently timing out when I try to connect to it at the moment. What could be causing this? This is a bit of a curveball.
The CPU usage is hovering around ~50% for all CPUs (according to top
). Values for wa
are between 0.0
and 3.0
. No swap memory is being used at all, and there's a ton of free mem available.
iostat
shows an %iowait
value of 0.51
. Other stats are:
Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
sda 4.88 1.02 2136.25 12365497 25895371840
sdb 0.00 0.00 0.00 9456 0
sdc 0.95 0.00 452.44 4781 5484405440
Writes/second are high - this is a write/heavy application. iotop
shows writes coming from the pgbouncer
process (a postgresql connection pooler), from async task queues and from nginx worker processes (probably writing to the access log). I don't see anything above 6% in the IO>
column - and most rows have 0.00%. SWAPIN
is 0.00%
throughout.
In short, CPU utilization isn't through the roof, memory utilization isn't the problem, and there aren't signs of excessive I/O related waiting going on. Why would the web app be infinitely loading/timing out when I try to access it? Could it be issues in sysctl.conf
or with my webserver? Need an expert opinion here.
The server in question is Ubuntu 14.04 LTS. Nginx is the webserver, used as a reverse proxy with Gunicorn (Django-based web application). The back-end is Postgresql 9.3, and Redis is in play as well. The DB resides in a separate VM.