we are running a mongodb replicaset on three machines. All three machines have around 16GB but only 255MB Swap. Swappiness is left on it's default value 60. The machines are running CentOS 6.4. The databases are much larger than the 16GB, but that's ok for us. The really working set is much smaller.
The problem we are facing is that the primary consumes eats up all available memory and than getting OOM-Killed. I know that this is the way how mongodb manages memory.
After the server is getting OOM killed someone has to manually restart it.
Is there any way to prevent mongodb from getting OOM killed? Adjust the swappiness? Increase swap space? I think that those settings will only increase the grace period before mongod gets killed.