Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

We're having an issue where one our Linux boxes (Ubuntu 10.04 LTS, running on EC2 with a quadruple-large size, 68GB of RAM and 8 virtual cores with 3.25GHz each) freezes up every few seconds. Typing in an ssh session will freeze, and running strace on one of the Postgresql processes that's running usually shows:

02:37:41.567990 semop(7831581, {{3, -1, 0}}, 1

for a few seconds before it proceeds (it always gets stuck at that semop).

OProfile shows that most of the time is spent in the kernel (60%) versus 37% in Postgresql.

The result of these halts (which began suddenly a day ago) is that load on the box has gone from 0.7 to 10+, and causes our entire stack to slow done.

Any ideas on how to track down what's going on? iostat doesn't show the disks being particularly slow or overloaded, and top shows user cpu % spike from 8% to about 40% whenever these back-ups happen.

share|improve this question
    
"halts" is kinda misleading, BTW. ;-) –  poige Feb 1 '11 at 6:55
    
are you sure you are running a PAE kernel? –  Hosm Feb 25 '11 at 1:58
    
Have you tried rebooting? :) –  AJ. Feb 25 '11 at 2:09
    
rebooting? it's not windows –  The Unix Janitor Mar 12 '11 at 10:29
    
Could it be the network/ing that freezes up? I noticed the basic EC2 instances had intermittently sluggish interactivity when I tried the Beta back in 2005. What I was seeing affected both ssh sessions and basic dynamic web pages, like perl CGI. –  Paul Mar 14 '11 at 7:55

7 Answers 7

We finally tracked it down to a PostgreSQL setting: "work_mem", which sets up how much RAM each Postgres process gets to do its sorts. We were spilling over the (tiny) default, which made the system hit disk, which is the kiss of death on EC2 (and the sudden spike in disk activity was freezing up the kernel in quick bursts of iowait).

share|improve this answer

Since you already located most of the time spent in the kernel, I'd suggest enabling CONFIG_LATENCYTOP and running, well, latencytop to see more. Can be done with oprofile, too, but latencytop is way more convenient.

share|improve this answer
    
Thanks, didn't know about latencytop! –  Zeppomedio Mar 25 '11 at 17:22

Check your available entropy when this is happening:

cat /proc/sys/kernel/random/entropy_avail

Ubuntu seems to have a bad habit of requiring real random numbers from the system when it doesn't need to, which can cause situations like this. Try and get the hardware random number generator working, it makes the issues go away, if you have it.

share|improve this answer

Take a look at this question Linux with 256GB of mem / 48 Cores - Machine starts thrashing/choking with tons of memory left. and see if the link about mysql and swap insanity with large memory helps.

share|improve this answer

We ran across a similar issue (different in that the pauses were minutes apart) when we deployed our Oracle servers on servers with 96Gb memory for the first time. We ended up tracking it down to the kernel process in charge of identifying memory that could be paged out. Setting the process to check smaller chunks more often took care of the problem.

share|improve this answer

Taking into consideration "68GB of RAM" I suspect it's related to VM inefficiency. Have you tried restarting Postgresql or rebooting?

share|improve this answer

I suspect your system is running out of semaphores. Check ipcs -l for current settings. Here's some info about tuning semaphores for postgresql. In particular I would try increasing the maximum number of semaphores system-wide (SEMMNS) and the maximum number of semaphores per set (SEMMSL). You can use sysctl -p to modify these settings.

share|improve this answer
1  
If the system ran out of semaphores, it would get an error - not a performance problem. –  Magnus Hagander Jan 26 '11 at 9:50
    
For reference: pubs.opengroup.org/onlinepubs/007908799/xsh/semop.html paying particular attention to "Upon successful completion, semop() returns 0. Otherwise, it returns -1 and errno will be set to indicate the error." from what I can see this is occurring and as such I agree with Phil, w.r.t. Magnus's comment, I'd say an error is occurring and the daemon is retrying leading to the performance issue. –  Oneiroi Mar 16 '11 at 15:57

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.