With a 4Gb RAM server and Windows 2003, I've determined by trial and error that setting a virtual RAM limit of 1.6 seems to kill the runaway processes without causing the normal processes to restart unnecessarily.
Support suggested a formula of 1.9 - all running processes = virtual memory limit, is there any theory behind this?