Linux freezes every few seconds

Question

We're having an issue where one our Linux boxes (Ubuntu 10.04 LTS, running on EC2 with a quadruple-large size, 68GB of RAM and 8 virtual cores with 3.25GHz each) freezes up every few seconds. Typing in an ssh session will freeze, and running strace on one of the Postgresql processes that's running usually shows:

02:37:41.567990 semop(7831581, {{3, -1, 0}}, 1

for a few seconds before it proceeds (it always gets stuck at that semop).

OProfile shows that most of the time is spent in the kernel (60%) versus 37% in Postgresql.

The result of these halts (which began suddenly a day ago) is that load on the box has gone from 0.7 to 10+, and causes our entire stack to slow done.

Any ideas on how to track down what's going on? iostat doesn't show the disks being particularly slow or overloaded, and top shows user cpu % spike from 8% to about 40% whenever these back-ups happen.

Answer 1

I suspect your system is running out of semaphores. Check ipcs -l for current settings. Here's some info about tuning semaphores for postgresql. In particular I would try increasing the maximum number of semaphores system-wide (SEMMNS) and the maximum number of semaphores per set (SEMMSL). You can use sysctl -p to modify these settings.

Answer 2

We finally tracked it down to a PostgreSQL setting: "work_mem", which sets up how much RAM each Postgres process gets to do its sorts. We were spilling over the (tiny) default, which made the system hit disk, which is the kiss of death on EC2 (and the sudden spike in disk activity was freezing up the kernel in quick bursts of iowait).

Answer 3

Take a look at this question Linux with 256GB of mem / 48 Cores - Machine starts thrashing/choking with tons of memory left and see if the link about mysql and swap insanity with large memory helps.

Answer 4

Since you already located most of the time spent in the kernel, I'd suggest enabling CONFIG_LATENCYTOP and running, well, latencytop to see more. Can be done with oprofile, too, but latencytop is way more convenient.

Answer 5

Taking into consideration "68GB of RAM" I suspect it's related to VM inefficiency. Have you tried restarting Postgresql or rebooting?

Answer 6

We ran across a similar issue (different in that the pauses were minutes apart) when we deployed our Oracle servers on servers with 96Gb memory for the first time. We ended up tracking it down to the kernel process in charge of identifying memory that could be paged out. Setting the process to check smaller chunks more often took care of the problem.

Answer 7

Check your available entropy when this is happening:

cat /proc/sys/kernel/random/entropy_avail

Ubuntu seems to have a bad habit of requiring real random numbers from the system when it doesn't need to, which can cause situations like this. Try and get the hardware random number generator working, it makes the issues go away, if you have it.