At work we're in the process of moving our production apps to a VM infrastructure, and so far so good.
One of the ideas we're throwing around is related to our software deployment process. The idea is that when we deploy an app, instead of deploying to an existing VM, we would create a new VM, deploy the app to it, test it, and then once it's looking good add it to the load balancer.
One major advantage we're expecting is that this would help control the drift that occurs between our test and prod environment configurations. If we are blowing VMs away with each deployment then even if somebody makes a change an individual server, it won't last long. (Ultimately we're driving toward centralized configuration management though we think the VM approach still helps.)
Does anybody have experience or even thoughts on this? One challenge one of my coworkers raised was that we might need to move to a one-app-per-VM configuration so that we're not forced to bring down multiple apps on the same VM when deploying one of the apps. Another potential issue is having multiple VMs with the same hostname while we're doing the deployment.
Would be interested in hearing your experience and thoughts. This is pretty new territory for my organization.