We're having an issue where one our Linux boxes (Ubuntu 10.04 LTS, running on EC2 with a quadruple-large size, 68GB of RAM and 8 virtual cores with 3.25GHz each) freezes up every few seconds. Typing in an ssh session will freeze, and running strace on one of the Postgresql processes that's running usually shows:

02:37:41.567990 semop(7831581, {{3, -1, 0}}, 1

for a few seconds before it proceeds (it always gets stuck at that semop).

OProfile shows that most of the time is spent in the kernel (60%) versus 37% in Postgresql.

The result of these halts (which began suddenly a day ago) is that load on the box has gone from 0.7 to 10+, and causes our entire stack to slow done.

Any ideas on how to track down what's going on? iostat doesn't show the disks being particularly slow or overloaded, and top shows user cpu % spike from 8% to about 40% whenever these back-ups happen.

  • "halts" is kinda misleading, BTW. ;-)
    – poige
    Feb 1, 2011 at 6:55
  • are you sure you are running a PAE kernel?
    – Hosm
    Feb 25, 2011 at 1:58
  • Have you tried rebooting? :)
    – AJ.
    Feb 25, 2011 at 2:09
  • 1
    rebooting? it's not windows Mar 12, 2011 at 10:29
  • Could it be the network/ing that freezes up? I noticed the basic EC2 instances had intermittently sluggish interactivity when I tried the Beta back in 2005. What I was seeing affected both ssh sessions and basic dynamic web pages, like perl CGI.
    – Paul
    Mar 14, 2011 at 7:55

7 Answers 7


I suspect your system is running out of semaphores. Check ipcs -l for current settings. Here's some info about tuning semaphores for postgresql. In particular I would try increasing the maximum number of semaphores system-wide (SEMMNS) and the maximum number of semaphores per set (SEMMSL). You can use sysctl -p to modify these settings.

  • 1
    If the system ran out of semaphores, it would get an error - not a performance problem. Jan 26, 2011 at 9:50
  • For reference: pubs.opengroup.org/onlinepubs/007908799/xsh/semop.html paying particular attention to "Upon successful completion, semop() returns 0. Otherwise, it returns -1 and errno will be set to indicate the error." from what I can see this is occurring and as such I agree with Phil, w.r.t. Magnus's comment, I'd say an error is occurring and the daemon is retrying leading to the performance issue.
    – Oneiroi
    Mar 16, 2011 at 15:57

We finally tracked it down to a PostgreSQL setting: "work_mem", which sets up how much RAM each Postgres process gets to do its sorts. We were spilling over the (tiny) default, which made the system hit disk, which is the kiss of death on EC2 (and the sudden spike in disk activity was freezing up the kernel in quick bursts of iowait).


Take a look at this question Linux with 256GB of mem / 48 Cores - Machine starts thrashing/choking with tons of memory left and see if the link about mysql and swap insanity with large memory helps.


Since you already located most of the time spent in the kernel, I'd suggest enabling CONFIG_LATENCYTOP and running, well, latencytop to see more. Can be done with oprofile, too, but latencytop is way more convenient.


Taking into consideration "68GB of RAM" I suspect it's related to VM inefficiency. Have you tried restarting Postgresql or rebooting?


We ran across a similar issue (different in that the pauses were minutes apart) when we deployed our Oracle servers on servers with 96Gb memory for the first time. We ended up tracking it down to the kernel process in charge of identifying memory that could be paged out. Setting the process to check smaller chunks more often took care of the problem.


Check your available entropy when this is happening:

cat /proc/sys/kernel/random/entropy_avail

Ubuntu seems to have a bad habit of requiring real random numbers from the system when it doesn't need to, which can cause situations like this. Try and get the hardware random number generator working, it makes the issues go away, if you have it.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .