I'm seeing average load times of
load average: 12.41, 11.94, 11.59 on a linux-based machine serving a web application. It has 16 cores, so the load average isn't unmanageably high.
However, this web application is frequently timing out when I try to connect to it at the moment. What could be causing this? This is a bit of a curveball.
The CPU usage is hovering around ~50% for all CPUs (according to
top). Values for
wa are between
3.0. No swap memory is being used at all, and there's a ton of free mem available.
iostat shows an
%iowait value of
0.51. Other stats are:
Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn sda 4.88 1.02 2136.25 12365497 25895371840 sdb 0.00 0.00 0.00 9456 0 sdc 0.95 0.00 452.44 4781 5484405440
Writes/second are high - this is a write/heavy application.
iotop shows writes coming from the
pgbouncer process (a postgresql connection pooler), from async task queues and from nginx worker processes (probably writing to the access log). I don't see anything above 6% in the
IO> column - and most rows have 0.00%.
In short, CPU utilization isn't through the roof, memory utilization isn't the problem, and there aren't signs of excessive I/O related waiting going on. Why would the web app be infinitely loading/timing out when I try to access it? Could it be issues in
sysctl.conf or with my webserver? Need an expert opinion here.
The server in question is Ubuntu 14.04 LTS. Nginx is the webserver, used as a reverse proxy with Gunicorn (Django-based web application). The back-end is Postgresql 9.3, and Redis is in play as well. The DB resides in a separate VM.