We're having a problem with our production website where our apache instances are mysteriously choking. The server never completely crashes, it's just that apache runs out of resources.
Looking at the situation, I've tracked it down to a whole bunch of our Apache processes getting stuck forever in flock_lock_file_wait mode. Once a process gets stuck in that mode, it's never getting out. So then the master process spawns another child, which eventually gets stuck in that mode again, until we finally run out of available children and the site grinds to a halt.
What I've found by using lsof on one of these 'stuck' processes is that they're stuck on deleted PHP5 session files. Here's some output from lsof on a stuck process:
apache2 32489 www-data 36u REG 8,1 195 5974866 /var/lib/php5/sess_bd5afe5ad8b05531d48a7e6f81638faa (deleted)
I go into that directory, and yes, that file is gone. So the Apache process is stuck trying to write (or read) session data from a deleted file.
My question is -- what the heck do I do now? This seems like something where Apache should somehow know not to get hung up, but I don't know how to tell it to just move on without rebooting Apache, which we're having to do almost every day now.
Any thoughts on where I can go from here?