Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We're having a problem with our production website where our apache instances are mysteriously choking. The server never completely crashes, it's just that apache runs out of resources.

Looking at the situation, I've tracked it down to a whole bunch of our Apache processes getting stuck forever in flock_lock_file_wait mode. Once a process gets stuck in that mode, it's never getting out. So then the master process spawns another child, which eventually gets stuck in that mode again, until we finally run out of available children and the site grinds to a halt.

What I've found by using lsof on one of these 'stuck' processes is that they're stuck on deleted PHP5 session files. Here's some output from lsof on a stuck process:

apache2 32489 www-data 36u REG 8,1 195 5974866 /var/lib/php5/sess_bd5afe5ad8b05531d48a7e6f81638faa (deleted)

I go into that directory, and yes, that file is gone. So the Apache process is stuck trying to write (or read) session data from a deleted file.

My question is -- what the heck do I do now? This seems like something where Apache should somehow know not to get hung up, but I don't know how to tell it to just move on without rebooting Apache, which we're having to do almost every day now.

Any thoughts on where I can go from here?

share|improve this question
    
I suspect that the lsof output is just what files the process happen to have an handle on. It could well be that the session file existed at the time it got stuck, but has since expired (and the file deleted) by the garbage collection process. Can you strace -p <stuck pid> and paste in the output? –  SmallClanger Sep 22 '11 at 15:35
    
Reboot? Why don`t you gracefully restart the httpd instead? –  Nils Sep 22 '11 at 21:07
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.