I've been running MongoDB for several years under the MMAPv1 storage engine, but now need to upgrade to WiredTiger. (Mainly so I can use event streams, but also because mmapv1 has been deprecated).
My application uses a handful of collections for every customer which means a LOT of collections. Due to the namespace file limits of MMAPv1 these are split across multiple databases. Sharding isn't necessary because although there are many collections, they are not large. Basically, this works fine and is stable.
However, switching to WiredTiger in my dev environment (which is Mac OS) I notice that not only does WiredTiger use a separate file for every collection and every index, but starting the
mongod process I see that ALL FILES ARE OPEN before any collections are accessed.
I estimate that migrating my production environment (CentOS 7) to use wiredTiger I could end up with close to a million files, and this will grow.
I'm aware of the ulimit issue and have read the documentation. The advice says that the recommended ulimit is 64,000. It concerns me that this is so much lower than what I'll end up with.
So my questions ...
Is it normal that ALL files are open at once? or does it sound like I'm doing something wrong?
Is this only a Mac issue? (I'm yet to test a migration on Linux)
If I set a ridiculously high
ulimit, would this impact the server in any negative way?