Despite the network config that led to the situation, this remains a legit question: what's the best way to get an ESXi host to see its iSCSI storage again without a reboot?
Note: I wasn't able to actually get this to work because my entire network stack was FUBAR and I needed to reboot ESXi to get it back anyhow. I'm posting this as an answer because what I found might be useful to those finding this question in the future.
The most useful link is this VMware KB article titled Cannot remount a datastore after an unplanned permanent device loss (PDL) (2014155)
The "won't come back" seems to be a feature intended for data protection. Any VMs that were using the storage that vanished are done, however temporary the "permanent" device loss may have been. Any VMs which were using the storage that vanished must be shut down (if possible) or powered off (more likely).
Before trying to get your ESXi server to see the storage again, ensure that it's actually there via the process in the VMware KB article titled Troubleshooting iSCSI LUN connectivity issues on ESX/ESXi hosts (1003681) You might just discover that your entire network stack is FUBARed in the process. I know I did...
From the first VMware KB article:
(Note: for iSCSI the devices may not be "naa" IDs and will instead look like "t10.IET_". I highly recommend cut and paste instead of trying to type those monstrosities by hand.)
The command to find the IDs was unhelpfully buried in the middle of the original article-- here it is for your convenience since you're undoubtedly going to need it to find the proper argument for the -d
option in esxcli storage core device world list -d <ID>
:
# esxcfg-scsidevs --uids
This shows the "t10" IDs like this:
Primary UID Other UID
mpx.vmhba45:C0:T0:L0 vml.0005000000766d68626135353a313a31
naa.60022190becbe0003b3e818108cdf2d5 vml.0200000000600221c0becce0001b3b918108cdf2c550455e4320c6
t10.IET_____000100000000000000000000000000000000000000000000 vml.010c000000202020202020202020202020202020202020202020202020202020202020626561663e30436e6e7f726f
t10.IET_____000100010000000000000000000000000000000000000000 vml.0100010000202020202020202020202020202020202020202020202020202020202020626561693131464952545541
Run this command to see the world that has the device open for the
LUN:
# esxcli storage core device world list -d <t10_id>
For example:
# esxcli storage core device world list -d t10.IET_____000100010000000000000000000000000000000000000000
You see output similar to:
Device World ID Open Count World Name
------------------------------------------------------------ -------- ---------- ----------
t10.IET_____000100010000000000000000000000000000000000000000 2060 1 idle0
If a VMFS volume is using the device indirectly, the world name
includes the string idle0. If a virtual machine uses the device as an
RDM, the virtual machine World ID is displayed. If any other process
is using the raw device, the corresponding information is displayed.
Notes:
If the host is not responding, run the command esxcfg-scsidevs –-list
to get the corresponding datastore name. Ensure all
virtual machines registered on the volume in a PDL state do not
require any further steps. If you have a virtual machine in that
state, attempting to Retry or Cancel an operation will not return the
virtual machine world ID. Click Cancel as the Retry operation cannot
succeed unless the volume is remounted.
Run this command to list all virtual machines running on the ESXi 5.x
host and identify the virtual machine registered on that LUN:
# esxcli vm process list
To kill the virtual machine World ID, run this command:
# esxcli vm process kill --type=force --world-id=World ID
For example:
# esxcli vm process kill --type=force --world-id=12131
Rescan the storage using this command:
# esxcfg-rescan -u vmhba#
Run this command to see the device state:
# esxcli storage core device list -d <t10-id>
If the issue persists, reboot the ESXi 5.x host where virtual machine
was registered.