2

An operating system is installed in a virtual machine environment, such as VMWare or KVM. A network shared disk volume (e.g., iSCSI disk) is used as the OS system partition. Heavy traffic is running in this OS. If the shared disk volume can't be accessed temporarily for two or three minutes (because of network problems or other reasons), and it is online again after this short period, what will happen? Will the OS be crashed, or will it continue to run without data corruption?

I have tested my case with Linux guest OS. During the non-access period, the Linux desktop hangs and I can't operate it. But when the system volume can be accessed again, I can use the desktop again, and find the previous tasks continue to run.

Although my test seems successful, I can't make sure that it is always OK. I know OS will retry IOs so maybe it doesn't matter if the disk doesn't return IO for a short period. But OS also uses swap partition to swap out some pages in memory. If the swap operations are paused because of disk, are there any serious consequences?

4
  • My gut feeling on this is it will occasionally and unpredictably fail. Might get better answers from serverfault
    – Robb
    May 5, 2011 at 9:58
  • So the OS is installed on the shared disk? May 5, 2011 at 9:58
  • VMWare can use iSCSI disk to install guest OS. Although I only let only one VM to access the disk, we can think that the OS is installed on the shared disk, because other hosts can still use iSCSI to connect it.
    – flypen
    May 5, 2011 at 14:17
  • Are you running hardware iSCSI or software iSCSI? If HW iSCSI, What kind of HBAs?
    – ahill
    May 15, 2011 at 23:45

6 Answers 6

4

The default timeout for scsi disks is 30 seconds, but you can change it by changing /sys/block/disk/device/timeout for example using echo 180 > /sys/block/disk/sda/timeout will increase the timeout to 180 seconds.

1

If I understand correctly, the guest OS system partition is local from the point of view of the OS, and it's only remote for VMWare?

Without knowing for sure, my experience with VMWare is that it will rather pause the virtual machine during this time. Actually I had a problem with VMWare ESXi where the storage (containing all virtual machines, and it was local!) was full when increasing the size of a growable partition. All VMs were paused. I had to delete a snapshot to free some space, though I'm not sure if they continued to run directly afterwards (or after a reboot). Wasn't a critical server though, and I'm just a developer, not a system administrator :)

1
  • It's a remote volume of VMWare machine, but it is local from the point of view of the OS. The VM OS think it as a local SCSI disk.
    – flypen
    May 5, 2011 at 14:18
1

If you lose the disk for 2-3 second, you probably be ok and the OS will continue one after it is available again. Although it will bitch and moan loudly in the logs.

If you lose the disk for minutes the OS may or may not Kernel Panic/BSOD but unless you are really really lucky you are going lose data and the system WILL become very unstable.

Yes the I/O subsystem will retry ... but it won't retry for minutes.

1

I imagine this will depend quite a bit on the virtualisation layer. FWIW, I just tested this using VirtualBox and it simply froze, which for all intents may as well be a crash. I don't have other systems to test on, nor do I believe that this behavior will be consistent. I suspect it will depend a bit on what the OS is actually doing at the time the connection broke.

1

This is quite a complex question really, and the answer depends on your host configuration. First of all the iSCSI layer has it's own timeout periods and retries. The same goes for the device-mapper-multipath that controls the block devices, and above that you have the QEMU disk layer and the disk controller driver within the guest OS. Not to get into too many details, if you anticipate using unstable storage, it is much safer to keep risks to the minimum. This can be achieved by disabling the QEMU disk caching function ( cache=none in the cmd line), and using werror=stop to make the guest pause whenever it hits an IO error, instead of trying to push that IO indefinitely. If you don't use these, with an unstable storage you are risking image corruption and data loss, though in some cases, if the guest OS detects the IO error (if you use propagation for example), it might simply remount it's FS in r/o mode.

In any case, it is generally better to avoid disk access bottlenecks, especially when VMs are involved. Multiple paths and separate networks for iSCSI traffic only are the common means of achieving this.

1

It depends on many settings. The OS is designed to retry I/O for a while. How long depends on the OS and settings of its I/O subsystem and all the layers below it.

For example, consider a linux VM running on VMware ESXi. The Linux VM thinks it is running on a SCSI disk, which is actually a VMDK file on a VMFS filesystem managed by VMware. The VMFS filesystem is actually located across the network on an iSCSI LUN on a SAN. Many layers, each with their own settings and timeouts. In this case you have to check the timeouts on both VMware's iSCSI initiator and Linux's SCSI subsystem.

In such a layered system it's smart to increase the default timeouts since there is a bigger chance of something failing temporarily. VMware actually takes care of some of this by itself. The VMware software iSCSI initiator has reasonably long timeouts as far as I know. Linux's default timeouts are a bit short:

$ cat /sys/block/sda/device/timeout 
30

Once you install VMware-tools on the VM, it takes care of raising the timeouts of the virtual disks to a safer value of 180 seconds. I'm not sure which value it sets for Windows VM's.

A longer timeout is no guarantee though. A guest OS with high disk I/O activity may not be able to tolerate sustained read and/or write requests for the duration of the timeout value. Windows guests may freeze or BSOD. Linux guests may go read-only on their root volumes which requires a reboot to fix.

While the OS may survive the disk I/O interruption, application(s) running on the OS platform may not. Applications themselves implement response timeout values which are likely going to be hard coded and non-configurable by a platform or virtualization administrator in the application itself.

A personal experience: I once upgraded my SAN firmware and rebooted the SAN. This reboot is fast enough to fall within the timeouts of both VMware ESXi and my Linux and Windows VM's. Usually all VM's kept running fine. However, this time a single VM didn't like the delay and crashed hard. No response whatsoever. So hard that I was unable to kill the VM and had to reboot the entire VMware host.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .