2

We're having a problem with our production website where our apache instances are mysteriously choking. The server never completely crashes, it's just that apache runs out of resources.

Looking at the situation, I've tracked it down to a whole bunch of our Apache processes getting stuck forever in flock_lock_file_wait mode. Once a process gets stuck in that mode, it's never getting out. So then the master process spawns another child, which eventually gets stuck in that mode again, until we finally run out of available children and the site grinds to a halt.

What I've found by using lsof on one of these 'stuck' processes is that they're stuck on deleted PHP5 session files. Here's some output from lsof on a stuck process:

apache2 32489 www-data 36u REG 8,1 195 5974866 /var/lib/php5/sess_bd5afe5ad8b05531d48a7e6f81638faa (deleted)

I go into that directory, and yes, that file is gone. So the Apache process is stuck trying to write (or read) session data from a deleted file.

My question is -- what the heck do I do now? This seems like something where Apache should somehow know not to get hung up, but I don't know how to tell it to just move on without rebooting Apache, which we're having to do almost every day now.

Any thoughts on where I can go from here?

2
  • I suspect that the lsof output is just what files the process happen to have an handle on. It could well be that the session file existed at the time it got stuck, but has since expired (and the file deleted) by the garbage collection process. Can you strace -p <stuck pid> and paste in the output? Sep 22, 2011 at 15:35
  • Reboot? Why don`t you gracefully restart the httpd instead?
    – Nils
    Sep 22, 2011 at 21:07

0

You must log in to answer this question.

Browse other questions tagged .