How to avoid massive number of files open at once?

Question

I've been running MongoDB for several years under the MMAPv1 storage engine, but now need to upgrade to WiredTiger. (Mainly so I can use event streams, but also because mmapv1 has been deprecated).

My application uses a handful of collections for every customer which means a LOT of collections. Due to the namespace file limits of MMAPv1 these are split across multiple databases. Sharding isn't necessary because although there are many collections, they are not large. Basically, this works fine and is stable.

However, switching to WiredTiger in my dev environment (which is Mac OS) I notice that not only does WiredTiger use a separate file for every collection and every index, but starting the mongod process I see that ALL FILES ARE OPEN before any collections are accessed.

I estimate that migrating my production environment (CentOS 7) to use wiredTiger I could end up with close to a million files, and this will grow.

I'm aware of the ulimit issue and have read the documentation. The advice says that the recommended ulimit is 64,000. It concerns me that this is so much lower than what I'll end up with.

---

So my questions ...

• Is it normal that ALL files are open at once? or does it sound like I'm doing something wrong?

• Is this only a Mac issue? (I'm yet to test a migration on Linux)

• If I set a ridiculously high ulimit, would this impact the server in any negative way?

Answer 1

Generally, mongodb use pool of connection, You can limit the number of incoming connections using the net.maxIncomingConnections run-time option. By restricting the number of incoming connections you can prevent a cascade effect where the mongos creates too many connections on the mongod instances.

Regarding your questions my answers inline:

Is it normal that ALL files are open at once? or does it sound like I'm doing something wrong? My Ans: it is not wrong process make persistent connections(Pool) since making connection is costlier operation.

Is this only a Mac issue? (I'm yet to test a migration on Linux): My Ans: It will work on same way only changes would be limits for each environments.

If I set a ridiculously high ulimit, would this impact the server in any negative way? My Ans: Once change the limit it surely impact system resource consumption, try to get number of connection require by your application with open threads and tune settings accordingly.

Hope above answers will help you.