You can have a production environment running on a single MongoDB instance. Under very special circumstances. Let us see what those circumstances are after we cleared up some things about replica sets.
On replica sets
Contrary to popular belief, replica sets have only one main purpose: ensuring the availability of the databases. Let us assume you have only a single instance. Each and every maintenance work, every server crash, every mistake in administration will cause a downtime. Downtime which affects not only your SLAs (bad enough), but may very well cause a DBA to get rung out of the bed at 06:00 am in the morning after a party night, and now he tries to get his caffeine level high enough to be able to restore the database(s) halfway drunk.
Furthermore: inevitably, you will loose all the data between your backup and the restoration of the service. First and obviously, you will loose all the data between the last backup and the point at which the server became unavailable. Then, you will loose all the data which would have been generated during the downtime.
Now let us assume you have a replica set with two data bearing nodes and and arbiter. Slightly better. Your primary fails, the other data bearing node gets elected and thanks to automatic failover – which most drivers provide – your service continues to run, without downtime and data loss. But: you have lost redundancy. So in order to reduce the risk, one DBA gets rung out of bed again, who now has to promote the arbiter to a data bearing node, wait for the data to be synced while hoping that the sync is faster that the change rate of your data (to be more precise: you hope that your replication oplog window is bigger than the time needed for syncing the data). If not, the data sync will fail and you have to shut down the application in order to let the sync succeed. What you have won with this setup is that you can choose when to shut down the application to restore redundancy.
Side note: If your data change rate exceeds the replication oplog window, you should shard. Always see to it that your replication oplog window is big enough.
Now let us assume you have three data bearing nodes, as suggested. Even when one server fails (or is updated etc), you still have redundancy. One node fails during night? Sleep tight!
So when can I have a production environment with a single server?
Taking the above into account, you can have a single instance production environment if, and only if you
- Have no SLAs and/or your customers can live with extended downtimes
- Your DBAs are up to being on call-standby to restore the service
- You and/or your customers can live with loosing data between the last backup and the restoration of the service.
In most serious business applications I know, the answer to one or more of those conditions is "No".
Can I have a production environment with two data bearing nodes and an arbiter?
Yes, under the assumption that you can live with the fact the losing one data bearing node will cost you redundancy. And that in this case you most likely have to resync your data, which requires you to closely monitor your replication oplog window and to make sure the time needed to resync fits it.
Given the price difference between an arbiter instance and a data bearing node, it is a question of risk management wether you choose between a setup with two data bearing nodes and an arbiter or the suggested setup with three data bearing nodes.
Can one reasonably use one ec2 instance as a production mongo server for a small database?
To put it bluntly: Not imho. It would be bordering negligence, increases risks which can be mitigated for comparatively little money and is most likely way more expensive than having at least a replica set with two data bearing nodes and an arbiter. If you take everything into account.
Does this need to be a 3 server replica set as indicated in the mongo docs?
Unless you really, really, really know what you are doing: Yes.