I have a Linux machine which runs ~10 in house written processes.
Every other day(!) the machine completely runs out of RAM, goes into swap and becomes unresponsive. This happens quickly over a period of a couple of seconds, so it's not feasible to sit watching the machine until it dies. It's a sudden leak, not a gradual one, so top(1) doesn't give any indications something bad may about to happen.
What is the best way of identifying which process(es) are causing the trouble?