We have 6 ESX servers running +150 VMs. Currently our VCenter server is one of these VMs. The other day we had a hardware failure in our DC (caused by a naughty UPS) which took out two of these servers. The first server it took out was running our primary VCenter server, the second running our HA/Heartbeat VCenter server, thus none of our hosts migrated off our two failed hosts onto the 4 working ones and we lost most of our VM management (users all use VSphere). This is a very unfortunate circumstance, and hopefully shouldn't happen too often, but I was wondering, is it a good idea to run our primary VCenter server on a separate box in a different datacenter*/redundant block dedicated to just VCenter, with the backup being a VM? Is it even possible? (All we have is the virtual appliance, though if it's available I wouldn't have thought it's too hard to track down).
*I'm ashamed to say, we run all our VMWare servers in a single DC. We mirror the SAN to a second DC but we have no servers there. They are only development/non-critical servers but people still shout if they're down.