We posted this same question on the Mongo forum, and the Mongo CTO responded, saying to review his presentation on how to optimize indexes
In this presentation, Mr. Horowitz states explicitly that sharding/horiz scaling can be overkill in many situations, and that design approaches (including some rather non-intuitive approaches that are kind of specific to Mongo) can make a given server scale much farther.
This presented some interesting concepts, including using client side logic to optimize how the db is used in a number of "non normalized" ways. There is a clear subtext to the presentation to the effect "if you just build by the book, you can easily hit unwanted problems related to scaling." For example, Mr. Horowitz (the CTO of 10Gen, maker of MongoDB) presents a "hybrid" design in which instead of one document per "record" you put perhaps 100 "records" in a document, resulting in a "bucket" kind of approach. This is done explicitly to reduce the index footprint. This is something that is coded on the client, and is not a "feature" of MongoDB. This hybrid approach may work for us, and could give us a 4x or 8x improvement in index size.
He also discusses "right balanced" btrees, which is basically optimizing the index design so that most queries access only the "right hand piece" of the index (as opposed to random access across the index, which, to perform well, requires that the whole index fit in RAM). This approach will not help us, as we need to query all over the index.
We are going to use these concepts as part of a review of our system.
(Interesting that of all the places I posted this question, the only person with a constructive response is the CTO of MongoDB itself.)
We found Mongodb, ultimately, to not be appropriate in a production env. Every couple of months, it dumps/trashes the entire db, and all data is lost. (It is not a primary data source, so we can live with the problem, though not happily.)
We have now completed a project to move to the elasticsearch stack, and are rolling that to production now. (We have used elasticsearch successfully for years.) Elasticsearch data is not as durable, as, say, Microsoft SQL Server, (we have had elasticsearch clusters fail with unrecoverable data loss), but elasticsearch is at least 10x (experientially, more than 100x) more reliable than Mongodb. (Elasticsearch, intelligently, makes no pretense of supporting Windows as a production platform, one of the big sins of Mongodb.)
We expect to have purged our entire env of Mongodb over the coming weeks.
The elasticsearch stack has delivered. We have found it to be very reliable, very scalable, and have not looked back at all. Mongo smelled great at the time, but it is gone for years now, and good riddance to it. We have been running two elastic clusters, one for log data (which replaced our Mongo server), and a second for real application data. Each cluster has 1-2TB of data. It took a lot of architecture work (particularly on the application side) to get elastic to both scale and perform, but deliver it does.