3

I'm trying to ensure that the revert process is as reliable as possible in our lab manager environments. We frequently (daily) revert a 10-12 server workspace to a previous version, upgrade it, test on it. Every few weeks, I hit a new problem after a revert, and a team or two has to wait while I monkey around.

The servers are Win2K3 servers running various applications. They are members of an domain, external to the workspace.

Question: What do you do to achieve 100% reliable revert? Any suprises in store for me, beyond the following? Better resolutions to these problems?

Note: Unfortunately, fenced workspaces are not practical in this scenario. These are unfenced environments. Saving the workspace to a to configuration and cloning tends to be far too slow for daily use - even though we've made disks as small as practical (10GB per machine).

Snags hit:

  • Machine changes password (?) or some other credentials with DC every x weeks. The snapshot can't connect to domain. Prevent by

    [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Netlogon\Parameters] "DisablePasswordChange"=dword:00000001

  • Computers revert from snapshot with incorrect times, times out of sync between machines. Chaos ensues. Final resolution: Make sure host is running NTP client - one of ours was not, ensure clients sync to host. Per VMWare, that was the root of many of our problems.

Summary good answers

3 Answers 3

3

I would recommend taking your snapshots when the VMs are powered off.

Even though vmware allows you to snapshot a live server, if you don't snapshot the memory it will be like a power failure has occured - and if you DO snapshot the memory you will have problems with things like NTP not running immediately.

The nice thing about vmware snapshots is that reverting is very fast. They don't so much make a "snapshot of the server" in the traditional sense - rather they start a new file containing any changes made to the disk AFTER the snapshot was taken. Hence, reverting means just discarding the "delta" file. (the drawback is that if you want to keep a snapshot around for a long time - the delta file continues to grow)

1
  • A good point. Forgot that I destroyed an IIS 6 metabase this way a few months back. Jun 10, 2009 at 19:57
1

If you sync VM time to host and host to DC, then it may not be a problem.

You can set vmware tools to execute scripts after resume & co. Create script to clean up : ipconfig /flushdns ipconfig /renew net stop w32time && net start w32time

You may use klist from the resource kit to delete/puge kerberos ticket and so to get a new one against your DC

1

I'd recommend running the DC in your configuration if at all possible. Get everything configured the way you like, power off the whole configuration, undeploy it and capture it to the library. If you capture the machines while they're running, you could run into errors (or be unable to) when you try to redeploy them if your environment is non-homegenous (i.e. trying to deploy a running machine that was started on an Intel processor to an AMD processor).

Rather then revert you'll just deploy a fresh copy of the entire configuration, and you could even deploy multiple fenced copies of this configuration.

Remember of course that all machines in the configuration have to be deployed to the same physical host ultimately.

3
  • Our simplified test environments are fenced. I agree it is much easier for administration in a general case. However, a lot of factors block this in the primary test environments: Mainly, the team of human QA uses it (cloning humans is probably illegal, whether or not you use SidGen to give them new identities). They want to be able to access the environments easily from their workstations, without the bother of RDP, lab manager consoles, multiple domains. If I had time to virtualize their full workstation with all the goodies they put on it, they might go for it. Jun 15, 2009 at 19:05
  • Hmm... not sure I'm understanding the requirements. If they're not using RDP or the lab manager console how are the accessing the machines? Why are you running them unfenced? Aside from the initial setup of the 12 machine configuration and capturing to the library, cloning a fresh copy to someone's workspace should only take a couple of minutes. Powering it on should take a bit longer, but not much.
    – Sage
    Jun 16, 2009 at 15:27
  • I agree, I'd rather we run fenced. But simply put, running unfenced for the convenience of QA. They don't wanna RDP - and convenience of many QA is more important than my convenience. That's another discussion. Jul 23, 2009 at 3:47

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .