1

I run ESXI 5.1 and all my virtual machines are in a NAS mapped by iSCI datastore.

My router broke and obviously all the vm's where cut from their disks.

What I understand is that when such event occurs, esxi stop all I/O from vm's and trie to reconnect to the datastore. If it can't after a certain among of time, it unmount the datastore as it may have been gone forever ...

I plugged a new switch then restart my esxi. I can see that the datastore appears online on the datacenter view from vCenter (but when I browse it nothing appears, I can't even make a new folder ...) and in my host's storage view the datastore doesn't appear... If I trie to add a new datastore to the host and I select iSCSI/LUN nothing appear either.

Beside, on the NAS (a Synology) I can see that there is no active connection on my LUN ...

So the big question is : How to get my datastore and all my vm's back ?

  • 3
    This begs the question - why is a router between your iSCSI initiators and their targets? – EEAA May 8 '14 at 20:13
  • @EEAA simply because the NAS is used for many other purpose and I have only 2 ethernet port ... used with port aggregation, so I can't connect directly to the NAS – 0x1gene May 8 '14 at 20:24
  • Well then you put a switch in between them (preferrably multiple switches). Not a router. – EEAA May 8 '14 at 20:25
  • Let me try to explain what @EEAA tries to say: You're using a SWITCH, not a ROUTER. (we hope) – MichelZ May 8 '14 at 20:26
  • @EEAA is it for performances issue ? Now I have only a switch left witch I use between them. I thought that putting it directly in the router would be better idea :/ but if there is only one device the issue would have been the same ... so yes better put 2 switch next time – 0x1gene May 8 '14 at 20:31
3

Connect to ESX using SSH/Local console, and try the following:

esxcfg-volume -l
esxcfg-volume -m <datastorename>

the first one lists your datastores and their status, the second one mounts it

  • Or just reboot. – ewwhite May 8 '14 at 20:14
  • Reboot might not work. – MichelZ May 8 '14 at 20:15
  • 1
    the command esxcfg-volume -l didn't return anything when I run it on my ESX – 0x1gene May 8 '14 at 20:21
  • That's bad then :) Have you tried the reboot? can you ping the storage? – MichelZ May 8 '14 at 20:22
  • I reboot many times, and the ping is OK – 0x1gene May 8 '14 at 20:25
1

Despite the network config that led to the situation, this remains a legit question: what's the best way to get an ESXi host to see its iSCSI storage again without a reboot?

Note: I wasn't able to actually get this to work because my entire network stack was FUBAR and I needed to reboot ESXi to get it back anyhow. I'm posting this as an answer because what I found might be useful to those finding this question in the future.

The most useful link is this VMware KB article titled Cannot remount a datastore after an unplanned permanent device loss (PDL) (2014155)

The "won't come back" seems to be a feature intended for data protection. Any VMs that were using the storage that vanished are done, however temporary the "permanent" device loss may have been. Any VMs which were using the storage that vanished must be shut down (if possible) or powered off (more likely).

Before trying to get your ESXi server to see the storage again, ensure that it's actually there via the process in the VMware KB article titled Troubleshooting iSCSI LUN connectivity issues on ESX/ESXi hosts (1003681) You might just discover that your entire network stack is FUBARed in the process. I know I did...

From the first VMware KB article:

(Note: for iSCSI the devices may not be "naa" IDs and will instead look like "t10.IET_". I highly recommend cut and paste instead of trying to type those monstrosities by hand.)

The command to find the IDs was unhelpfully buried in the middle of the original article-- here it is for your convenience since you're undoubtedly going to need it to find the proper argument for the -d option in esxcli storage core device world list -d <ID>:

# esxcfg-scsidevs --uids

This shows the "t10" IDs like this:

Primary UID                                                     Other UID
mpx.vmhba45:C0:T0:L0                                            vml.0005000000766d68626135353a313a31
naa.60022190becbe0003b3e818108cdf2d5                            vml.0200000000600221c0becce0001b3b918108cdf2c550455e4320c6
t10.IET_____000100000000000000000000000000000000000000000000    vml.010c000000202020202020202020202020202020202020202020202020202020202020626561663e30436e6e7f726f
t10.IET_____000100010000000000000000000000000000000000000000    vml.0100010000202020202020202020202020202020202020202020202020202020202020626561693131464952545541

Run this command to see the world that has the device open for the LUN:

# esxcli storage core device world list -d <t10_id>

For example:

# esxcli storage core device world list -d t10.IET_____000100010000000000000000000000000000000000000000

You see output similar to:

Device                                World ID  Open Count  World Name
------------------------------------------------------------  --------  ----------  ----------
t10.IET_____000100010000000000000000000000000000000000000000      2060           1  idle0

If a VMFS volume is using the device indirectly, the world name includes the string idle0. If a virtual machine uses the device as an RDM, the virtual machine World ID is displayed. If any other process is using the raw device, the corresponding information is displayed.

Notes:

If the host is not responding, run the command esxcfg-scsidevs –-list to get the corresponding datastore name. Ensure all virtual machines registered on the volume in a PDL state do not require any further steps. If you have a virtual machine in that state, attempting to Retry or Cancel an operation will not return the virtual machine world ID. Click Cancel as the Retry operation cannot succeed unless the volume is remounted.

Run this command to list all virtual machines running on the ESXi 5.x host and identify the virtual machine registered on that LUN:

# esxcli vm process list

To kill the virtual machine World ID, run this command:

# esxcli vm process kill --type=force --world-id=World ID

For example:

# esxcli vm process kill --type=force --world-id=12131

Rescan the storage using this command:

# esxcfg-rescan -u vmhba#

Run this command to see the device state:

# esxcli storage core device list -d <t10-id>

If the issue persists, reboot the ESXi 5.x host where virtual machine was registered.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.