Is there any reason to NOT have a swap file on CentOS
You have some type of horizontal scaling in place which increases your number of servers based on their memory usage, thus eliminating the possibility of swap being used.
You have a specific performance requirement that means you cannot allow your program to use swap for processing as it is significantly slower than memory, so rather than use swap, you choose to monitor memory usage closely and increase memory as needed.
Will my system just die the first time it gets seriously loaded up with users and memory demands?
If you have no idea how much load your server is going to experience, and you don't have enough memory, and you have no swap space, your application will stop.
Will swap save me from that, but only by imposing an unpleasant performance hit?
What exactly happens when the application stops? Which application stops?
If I had 3 processes and 1 was leaking memory, would all 3 die or the 1 that was leaking die?
Would 1 force the other 2 to die?
Think of it like this:
If you make a program and tell the program to fill up a variable with an infinite amount of integers, very soon that application will encounter a memory error and it will exit.
It's the same. Whichever application is trying to access more memory for it's operations is going to fail. So if you have 1 byte left and your application creates a 32 bit integer, you need 2 bytes, the operation fails because there's not enough memory, and depending on how well the application handles memory errors, it will either fix itself or exit.
The same goes for 3 applications at once. If by coincidence they all request more memory when the system has only 1 byte of memory left, they should technically all fail at once.
Somebody can correct me on this if they see any errors.