I've spent hundreds of hours trying to track down the cause of a process that mysteriously terminates at random intervals on 64-bit CentOS 5, and I'm so far no closer to understanding why. We've looked at the OOM killer, looked at every possible log file, done deep postmortems on the server after the event, added debug code to trigger a core dump on any "unusual" termination event, etc.
The process in question starts normally, and will often run on a given server for long periods of time (days, weeks, sometimes longer), but at random intervals on random servers, with no apparent cause-and-effect, it will simply exit. No logs, no core file, no nothing.
I'm not sure what to do next--hoping to get some ideas for troubleshooting this that I haven't thought of.