Virtualization Pitfalls/Lessons Learned

Question

What are some pitfalls or lessons learned after converting existing hardware to a virtualized environment? Is there anything you tried to virtualize but will never do again?

Answer 1

ALWAYS eject any virtual media (CD/DVD/Floppy) once you've finished as failure to do so will often stop a vMotion in its tracks.

Get your NTP and DNS setup correctly, this will save you from contemplating suicide :)

You can never have enough memory, or storage.

Make sure you have remote, OS-less, access to your machines such as HP's iLO system.

Keep a repository of OS/App .ISO files.

Not a direct answer to your question but in the hope that someone saves themselves from tearing their hair out in the future by finding this response - HP blade servers don't ship with their 'VT'-bit enabled by default, you have to enable it in the BIOS (F9). Without this ESX 3.5U4 doesn't throw a useful error, no it just hangs prior to the code install :(

Answer 2

To answer the question as asked - pitfalls related to P2V migrations.

First off - P2V migrations work very well for the most part. The cleaner and newer the systems the better but even with migrating old (NT4 systems) my success rate after more than a hundred migrations in a range of environments has been around 90%. That's systems that have migrated and been handed over to production on the day (well mostly on the night) planned. I've only ever had one system that we had to reverse out of after an apparently successful migration - a SQL box that needed more CPU horsepower than the platform could ever deliver. VMware Converter is good and free (for the non enterprise version), Platespin is very good (but costly).

That said - there are things to avoid.

MSCS Clusters. You can make them work but it's never a great idea and Microsoft will almost certainly not help you in any way if you have problems later. Build new stand alone systems instead.

Large SQL servers - emphasis on large. These should have been red flagged from a CPU requirements POV in advance but don't be tempted to move one if you aren't certain that the target VM will have ample CPU headroom.

If you are planning to change system names or ip-addresses (or both) during the migration then first consider don't doing that and if you absolutely have no choice then make sure you have people on hand who understand how those changes might affect the systems in question. My worst migration ever was an RSA ACE server used for authenticating a DMZ located VPN where the client refused to listen to my objections and insisted on changing both name and ip-address during the migration.

Related to the above - if you have anything other than a completely flat network then build some test VM's and make 100% sure that your VM networks perfectly replicate the physical ones you are migrating from.

In Windows AD environments always make sure you have a local admin account on the box being migrated. And test it before you migrate.

Make sure you have a good idea of how long things will take. P2V copy times will vary depending on available network bandwidth (obviously) but also can be dramatically affected by the number of files in each volume being migrated. This is particularly a problem with Platespin migrating NT4* systems but will affect any P2V software copying at the file level (which generally applies if you opt to resize volumes). Copy rates of 70-80Megabyte a second are possible with GigE networks, relatively fast source and a good target setup but 20-30Megabyte/sec is more typical and for the aforementioned NT systems with 100Meg networks and lots of files I've seen copy rates drop down into the 50kilobyte/sec range.

• Ideally you would get rid of these but some people don't have that luxury and getting such an OS of the completely unrepairable hardware its probably running on is almost always a good idea.

Answer 3

• Have a solid backup strategy in place beforehand. Decide if you are going to back up the VM as if it were on bare metal, or if you are going to back up the virtual hard disks on the data stores (or both). Generally speaking, I found that my required backup footprint increased significantly at first, so be prepared for an initial spike where you might be backing up both an old physical machine and a new VM before you get all of the kinks worked out.
• VM sprawl is also something to watch out for. Once virtualization starts taking off, the urge to move everything to VM becomes great. While this can work, you probably didn't order exactly enough hardware right off the bat.
• I think there are machines that can't be converted, and other machines that probably shouldn't be converted. While it is nice to be able to take a 10 year old physical machine and clone it to a VM, with warts and all, there are certainly scenarios where you would be better off building a
  clean OS and migrating objects from the physical machine over. Sometimes you are better off not converting over the cobwebs.
• Be prepared to use a lot of network ports. If you have systems that run on different VLANS, while single ports could be trunked, you probably want to have individual ports for your VLANs feeding into your vSwitch. If you want redundancy, and you are using iSCSI, you could be looking at a lot of NICs.

Answer 4

From my experience, be VERY careful about your storage medium. We went with an iSCSI SAN that turned out to only support 100Mbit connections. Running one VM on the system wasn't bad, two was less the adequate... and by the time we reached our target of 8 VMs they were horrible.

My personal lesson learned: Check the rated IOPS and read more reviews about a product that relate to the way you intend to use the storage device

Another handy thing I've learned... Making a 'backup' disk image after a base install and hardening will quicken the build of any other system and is a very handy thing to keep around.

Answer 5

Try not to run production database servers in a Virtual Environment. The overheads for I/O is unacceptable. We had huuuge problems when our DBA allowed our primary MSSQL server to become virtualised. Queries were taking thousands of milliseconds to run. When we convinced them to move it back to a dedicated box, there was a 10,000% increase in throughput and speed.

Answer 6

Use redunant network for vmotion/vmkernel traffic. You don't want virtual machines shutting down just because a switch rebooted.

Oh, and leave one DC/DNS/DHCP-server out of the virtualization. Your users will hate you less if you get a major SAN crash.

Answer 7

In case you don't have one already - Have a complete backup of the physical machine prior to the migration. An image is probably best or and an ASR/system restore or whatever that gives you a complete system snapshot, instead of the usual content backup that most machines have.

P2V tools can backfire on you unexpectedly, ruining the physical machine (i had VMWare converter kill a machine i was trying to P2V once, luckily it was just a test migration). Be prepared to have to restore the system from scratch. Yes, this is maybe a 1000 to one chance, but do YOU want to be that one?

Answer 8

VMWare Converter creates virtual machines that boot from scsi. MS virtual machines cannot boot from scsi. [edit - apparently version 4 of the converter now lets you specify SCSI or IDE, I love those guys]

If you're going to virtualize a non-ACPI physical machine, buy some software for the purpose. (unless you have a couple of weeks for an exciting journey in discovery!)

Also, VMWare Converter will tackle jobs at which MS SCVMM will throw up its hands in dispair.

Bring a lot of RAM.

Don't do anything until the virtualization tools (whether VMWare or MS) have installed.

If you're going to move it to another platform/version, uninstall the aforementioned tools.

Mind your CPU limits. P2V of a 2 CPU windows 2000 taught me that only 1 is supported.

• 2000 - 1 core
• 2003 - 2 cores
• 2008 - 4 cores

Answer 9

If you're going to use SAN to store your VM images, make sure that you label your devices and hosts VERY CLEARLY. Removing host-to-disk mappings on the SAN does terrible things if they are still in use by VMs.

Answer 10

Microsoft won't support Exchange 2003 running in VMware (at least that was the officially response). With a lot of arm twisting we were able to get some unofficial support from them, but it caused extra headaches in an already stressful crisis.

Answer 11

A lot of these are VMware specific:

• If it performs poorly as a physical machine, it will perform poorly as a virtual machine.
• The Cold Clone ISO is your friend
• Garbage in, Garbage out. If you're P2Ving older systems they can leave an unnecessary footprint. See Viewing Ghost Hardware after P2V, Steps Required after P2V, and p2v-scripts.pdf
• Make sure your guest operating systems are supported by your P2V software.
• Windows 2000 can be a pain regarding SCSI drivers

Answer 12

Stupid annoyance with VMware: different versions of VMware use different SCSI drivers for their virtual disk devices. It's entirely possible to waste 2 hours before considering that option.

Answer 13

Well, so far I haven't got any horror stories about myself doing the virtualization. However, a few notes though.

1. Planing carefully in details ahead. Especially do some homework what can't be virtualized.

2. If vendor of the application running on your server doesn't support virtual environment, wait until they support.

3. Implement w/ a SAN as the storage storing all VM images.

4. Run ESX or ESX(i), or Hyper-V, to gain most performance.

Maybe more but that's all for now. :)

[update] here is another one. Apply the latest firmware to the host server. I had one that I didn't do, which gave me purple screen once a few days and crashed the server completely.

Answer 14

Virtualization impact is around 5% of performance of overhead. Measure resources consumption on existing environnement to determine if your virtualization environnement can take this load.

Before going live with your virtualization solution:

• Check that you know how to backup AND restore VM. Using snapshot may not be supported, as on Windows DC
• Ask your editor if its solution is supported inside VM. Microsoft maintain list of their supported software inside VM: KB 897613
• As it's very easy to create VM, people tends to generate a new VM for every request. Then you have more VM than your solution is planned to support.

Answer 15

Is there anything you tried to virtualize but will never do again?

I wouldn't say I'd not try it again, but layered virtualisation is not pleasant to deal with.

By layered I mean running xen or esx on virtualised hardware such as Egenera, HP Virtual Connect or Cisco UCS. It sounds like a good idea, but can be very time consuming to debug.

Answer 16

In VMWare, know where the snapshots endup. We had ours configured to end up in the LUN on the SAN with the VM files themselves. A tech was practicing the Snapshot process on a LUN that was almost full. Later he rebooted the VM for some reason, and the log files caused the VM to not start. It was a bit of luck that lead us to the LUN being full as the cause.

Answer 17

If you go with a dynamically expanding VHD then ensure you go big enough. If you go with 100GB and you end up only using 20... no biggie. However, if you went with 25 then you have some work ahead of you.