Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

This weekend we are planning on upgrading our IOS on our Cisco switch. This switch has our three ESX servers and the iSCSI SAN that the ESX server use connected to it.

In the past when we have found it necceary to reboot the switch all running VM's get shutdown, and then rebooted by the ESX hosts. I am trying to avoid this as it is not a graceful shutdown event.

Is there a configuration change that I can make in VirtualCenter that would prevent this from occuring? Or any other ideas as to why this would happen in it's not VirtualCenter related?

share|improve this question

4 Answers 4

up vote 2 down vote accepted

I know this answer comes quite a while after the questions, but I recently ran into this issue and found a completely different resolution to the reboot of VM's.

If you have a VMware HA cluster, then in VirtualCenter, Right-click on the Cluster, and choose Edit Settings.

In the "Cluster Settings" dialog box, choose VMware HA on the left. In the right hand pane, you will see a section titled Default Cluster Settings Set the Host Isolation response dropdown to "Leave VM Powered On"

That way, when the network connectivity comes back, the hosts are still there and can continue doing their jobs.

share|improve this answer
    
Perfect. Thanks Scott. –  Richard West Jan 28 '10 at 21:41

That sounds like you have a dodgy VMware HA setup. Mind you if you have all of your ESX networking plugged into a single switch then you have a network that needs to be re-designed a bit.

If you have an ESX Cluster (especially with HA enabled) then each host should have two separate service console ports connected to two separate physical nics that are plugged into two separate switches so that you never lose management access to the Hosts when you lose a single switch (or take it out for maintenance). If ESX hosts are configured in a HA cluster and they lose contact with each other's Service Console interfaces for longer than 15 seconds HA will attempt to restart protected VM's, and will (by default) shut down the running VM instance.

share|improve this answer
    
Yes I realize that the network needs some addtional redundency :-) It sounds like I should be able to disable the Host Monitoring at the cluster level to prevent the ESX servers from thinking the have become isolated (which they have, however I don't want them to try and reboot the VM's). –  Richard West Oct 23 '09 at 21:15
1  
The problem seems even simpler: ESX is losing its disk since it's iSCSI connected over a single switch and you've rebooted that switch. –  MikeyB Jan 28 '10 at 19:08

There is a timeout value for your storage hidden somewhere in the guest os. when windows cannot access it´s disk for more than x seconds it crashes.

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Disk
Name    TimeOutValue
Typ     REG_DWORD
Value   60 (dezimal) for 60 Seconds
share|improve this answer
    
I have some systems that have this timeout set to 120 seconds when we did a firmware update on the SAN which required a reboot. However I'm sure there are some systems that I have not set this value to yet. Is there a powershell script, or another way, that I could use to set this value remotly, though a script? –  Richard West Oct 23 '09 at 21:38
1  
You could do it in Powershell but the reg.exe commandline will do the trick. See technet.microsoft.com/en-us/library/cc742162(WS.10).aspx for details but this should do the trick: reg add \\VM-NAME\HKLM\System\CurrentControlSet\Services\Disk /v TimeOutValue /t DataType REG_DWORD /d 120 /f –  Helvick Oct 23 '09 at 22:40
    
Thanks Helvick! –  Richard West Oct 25 '09 at 14:54

Sorry if I'm being dim here but are you saying that you store your VMs on the iSCSI box and each host is only connected to one switch?

If that's the case then taking down that switch will absolutely 100% kill every VM stored on the iSCSI SAN box as each host will lose not only it's networking but also its persistent storage on the iSCSI SAN box.

The only way to survive this sort of change, bar building in redundancy, would be to Storage vMotion every VM to each hosts local storage for the duration of the switch outage and then back to the SAN when the switch is back up.

If this isn't your situation please could you add some additional details to clarify the situation.

share|improve this answer
    
Chopper3 - you are correct, however we have changed the disk timeout setting within each VM to 120 seconds. The switch takes about 75 seconds to reboot. This allows the switch to reboot before the VM guest OS start to panic. I know that this is not the ideal solution, but I'm dealing with the hand that I'm delt at the moment. The problem that I was having was the HA settings forcing a restart becuase all ESX servers are isolated from each other during that time period. Scott Pascoe's answer did the trick. –  Richard West Jan 28 '10 at 21:41

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.