My employer requires all email correspondence in the organization to be indexed, for regulatory compliance reasons. Our mail server is not managed by us, and does not have this feature. Each workstation has its own email software. Aside from looking for a new mail provider, how should I index emails in a mostly realtime fashion?
Honestly, indexing data of any sort from multiple places in multiple formats is a recipe for disaster. You would be better off setting up your own internal email server, switching to a provider that offers indexing features, or adding a proxy-email server are pretty much your best options. All three of those suggestions come with their own pros and cons.
If this is legitimately a business requirement... your boss will pay for it. If you have a small IT department... let someone else to the work for you. (read as "let office365, google, or other similar company be your email provider") Trying to do this yourself without a dedicated person for that task will just end up making you look like you don't know what you're doing... and will probably get you fired at some point.
I think you may be looking for a service or appliance such as the Barracuda Message Archiver. I use the appliance version and it indexes all mail as it is sent / received.
The simplest, rawest way to do it would be to tap your organization's internet gateway, listening for connections to the mail server and perform packet capture on all emails going inbound and out. Store the captured packets in some sort of noSQL db.
There are some open-source project that can do this (Moloch comes to mind) but you might have to get a bit creative if the mail server uses SSL.