Why is my Rails app under Passenger occasionally grinding to a halt?

Question

I have a Rails 2.3.2 app running on Apache with Passenger, and recently, the strangest thing has started happening.

Every so often, the app will just stop processing requests entirely and they'll all back up until they time out. Restarting Apache seems to be the only way out. The system is under relatively light load until this happens, at which point it's under no load at all.

I realize this isn't a lot to go on, but how can I even figure out what's causing this? Of note, global queuing is turned on; Ruby is ruby-enterprise-1.8.6-20080610; and Passenger is the version that came with the aforementioned Ruby Enterprise Edition.

Answer 1

One resource that I found useful was the section of the Passenger users guide where they talk about this:

If one of your application instances is frozen (stopped responding), then you can figure out where it is frozen by killing it with SIGABRT. This will cause the application to raise an exception, with a backtrace.

Answer 2

Could it be something in the code? Something waiting on input blocking the thread and causing multiple processes to just wait?

Answer 3

Since you don't know what's causing the problem, I suppose you should make sure you're logging everything at least. The last few log entries before a crash could reveal your culprit.

The issue could possibly be something in the code. Well, to find out, you could deploy a very simple rails app, so code wouldn't be an issue, but with the exact same setup (apache + passenger + REE), and send it requests with a little ruby script overnight. Run it on your personal computer and see if it crashes. Of course you have to be sure you've set up your environment the exact same way.

It's also possible that you're running out of memory, though that depends on the specifics of your app and system. This post comes to mind. You may want to try modifying REE's memory management settings as described in the post to see if it solves the issue (won't hurt to try in any case).

I would suggest posting this question on stackoverflo as well if you haven't already.

Answer 4

I've had the same thing, I was hosting several rails 1.2.3 apps using apache + fastcgi for a year without a single crash, then I switched to a new server with apache + passenger and they crash randomly. Apache just stops responding.

I've been using monit to save it until now, and my answer so far has been to migrate to a fresh server (virtual) with nginx instead of apache, and with the latest passenger and rubyee. So far so good!

The apache server is still crashing and its only serving a mongrel site now, so I would assume its some issue with apache or my specific config. Give nginx a go if you can :)

Answer 5

Another potential answer: RMagick has been known to cause problems with Passenger and smart spawning.

This thread from the Passenger mailing list has more information:

It looked like the issue may have been fixed in REE 20090113, but I think it is just happening more infrequently. I found a frozen process this morning running passenger 2.0.6 and REE 20090113 on Centos 4 64- bit.