I have a weird situation with my Hosted Engine in oVirt.
We have an oVirt cluster set up using gluster as the storage for the engines isos and all the information. About a week ago two of the three servers went down. We restarted the machines about three times and the the gluster hosts reported as connected, and did not report any split-brain errors.
The hosted-engine process tried to bring up the hosted engine on one of the hosts, but it went into an EngineUnexpectedlyDown state. And it would subract 1600 from the servers score and then try to bring it up on the next machine until they all ended up with about 800 score and it would just try to boot on one machine and then just sit there with a "failed to reach vm" message.
We've figured out that the hosted-engine is actually booting as we can connect to it wit ha vnc client. But it seems to be in some sort of locked state. If you log into the hosted engine, even as root no files can be changed and the vm in inaccessible through any other means.
Is there any way to see if the vm is locked/read-only? And is there anyway to remove said lock?