I think Linux can do this using ulimit, I'm looking for something similar in Win7 x64. The problem is some times a process doesn't know the physical memory is already exhausted and continues to allocate memory, which will slowdown the system to a crawl since system is allocating from pagefile. The only way to recover is hard reboot.
Thanks