I want to set the max limit on the physical memory an application can use in IIS 7. Should I set a limit on Private Memory Limit or Virtual Memory Limit? Can some one point me to documentation of all settings in IIS 7
Limit the Private Memory (Committed bytes) this is what corresponds to your actual Memory usage.
Virtual Memory isn't shared across applications and on x64 worker processes will generally reserve much much, more than they actually end up using. Forcing limits on this really only makes sense on x86.