Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Environment: Ubuntu 10.04 LTS, Passenger, Nginx 1.0.6, MySQL, Ruby 1.9.2, Rails 3.1

After some amount of time, the server ends up with a gradually increasing number of processes that are stuck at 100% CPU

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND 
 2393 avitus    20   0  496m 381m 1392 R  100  9.4  25:10.74 Rack: /home/web ...

Running a strace on any of the stuck PID's gives the following:

Process 2393 attached with 3 threads - interrupt to quit
[pid  2396] futex(0x8ca80e4, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid  2394] restart_syscall(<... resuming interrupted call ...>) = -1 ETIMEDOUT (Connection timed out)
[pid  2394] gettimeofday({1322590778, 346573}, NULL) = 0
[pid  2394] futex(0x821db60, FUTEX_WAKE_PRIVATE, 1) = 0
[pid  2394] clock_gettime(CLOCK_REALTIME, {1322590778, 346885177}) = 0
[pid  2394] futex(0x821db84, FUTEX_WAIT_PRIVATE, 33872659, {0, 9687823}) = -1 ETIMEDOUT (Connection timed out)
[pid  2394] gettimeofday({1322590778, 356921}, NULL) = 0
[pid  2394] futex(0x821db60, FUTEX_WAKE_PRIVATE, 1) = 0
[pid  2394] clock_gettime(CLOCK_REALTIME, {1322590778, 357196244}) = 0
[pid  2394] futex(0x821db84, FUTEX_WAIT_PRIVATE, 33872661, {0, 9724756}) = -1 ETIMEDOUT (Connection timed out)
[pid  2394] gettimeofday({1322590778, 367240}, NULL) = 0
[pid  2394] futex(0x821db60, FUTEX_WAKE_PRIVATE, 1) = 0
[pid  2394] clock_gettime(CLOCK_REALTIME, {1322590778, 367459723}) = 0
[pid  2394] futex(0x821db84, FUTEX_WAIT_PRIVATE, 33872663, {0, 9780277}) = -1 ETIMEDOUT (Connection timed out)
[pid  2394] gettimeofday({1322590778, 377586}, NULL) = 0
[pid  2394] futex(0x821db60, FUTEX_WAKE_PRIVATE, 1) = 0
[pid  2394] clock_gettime(CLOCK_REALTIME, {1322590778, 377807840}) = 0
[pid  2394] futex(0x821db84, FUTEX_WAIT_PRIVATE, 33872665, {0, 9778160}) = -1 ETIMEDOUT (Connection timed out)
[pid  2394] gettimeofday({1322590778, 387932}, NULL) = 0
[pid  2394] futex(0x821db60, FUTEX_WAKE_PRIVATE, 1) = 0
[pid  2394] clock_gettime(CLOCK_REALTIME, {1322590778, 388162450}) = 0
[pid  2394] futex(0x821db84, FUTEX_WAIT_PRIVATE, 33872667, {0, 9769550}) = -1 ETIMEDOUT (Connection timed out)

Including the 'c' flag for strace gives:

Process 2393 attached with 3 threads - interrupt to quit 
Process 2393 detached Process 2394 detached 
Process 2396 detached 
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 94.97    0.003172           2      1489       744 futex
  3.74    0.000125           0       745           clock_gettime
  1.29    0.000043           0       745           gettimeofday
  0.00    0.000000           0         1         1 restart_syscall
------ ----------- ----------- --------- --------- ----------------
100.00    0.003340                  2980       745 total

I can kill -9 the stuck processes and the application and server appear to carry on happily. I've run out of ideas on how to proceed with debugging so if anyone has any advice as to the cause or other avenues of investigation it would be great to hear.

share|improve this question
    
hello andy, have u find the reason of this problem? i get the same problem –  Neo Oct 8 '12 at 9:13

2 Answers 2

Try setting passenger_spawn_method to conservative in Passenger. I'm having this issue with Mongo and came across:

http://code.google.com/p/phusion-passenger/issues/detail?id=684

and:

https://github.com/rails/rails/issues/1339

I don't know why it's not working, but hopefully that will get you going if you haven't figured out the solution already.

share|improve this answer

That particular behavior (checking futex every 20 ms and then checking the time of day) appears to be the normal behavior for an idling Ruby process:

http://www.ruby-forum.com/topic/192255

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.