Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I experienced an odd issue this morning where an NFS issue appeared to have taken down the majority of my VMs hosted on a small vSphere 5.0 estate.

The infrastructure itself is 4x IBM HS21 blades running around 20 VMs. The storage is provided by a single HP X1600 array with attached D2700 chassis running Solaris 11. There's a couple of storage pools on this which are exposed over NFS for the storage of the VM files, and some iSCSI LUNs for things like MSCS shared disks. Normally, this is pretty stable, but I appreciate the lack of resiliancy in having a single X1600 doing all the storage.

This morning, in the logs of each ESX host, at around 0521 GMT I saw a lot of entries like this:

2011-11-30T05:21:54.161Z cpu2:2050)NFSLock: 608: Stop accessing fd 0x41000a4cf9a8  3
2011-11-30T05:21:54.161Z cpu2:2050)NFSLock: 608: Stop accessing fd 0x41000a4dc9e8  3
2011-11-30T05:21:54.161Z cpu2:2050)NFSLock: 608: Stop accessing fd 0x41000a4d3fa8  3
2011-11-30T05:21:54.161Z cpu2:2050)NFSLock: 608: Stop accessing fd 0x41000a4de0a8  3
[....]
2011-11-30T06:16:07.042Z cpu0:2058)WARNING: NFS: 283: Lost connection to the server 10.13.111.197 mount point /sastank/VMStorage, mounted as f0342e1c-19be66b5-0000-000000000000 ("SAStank")
2011-11-30T06:17:01.459Z cpu2:4011)NFS: 292: Restored connection to the server 10.13.111.197 mount point /sastank/VMStorage, mounted as f0342e1c-19be66b5-0000-000000000000 ("SAStank")
2011-11-30T06:25:17.887Z cpu3:2051)NFSLock: 608: Stop accessing fd 0x41000a4c2b28  3
2011-11-30T06:27:16.063Z cpu3:4011)NFSLock: 568: Start accessing fd 0x41000a4d8928 again
2011-11-30T06:35:30.827Z cpu1:2058)WARNING: NFS: 283: Lost connection to the server 10.13.111.197 mount point /tank/ISO, mounted as 5acdbb3e-410e56e3-0000-000000000000 ("ISO (1)")
2011-11-30T06:36:37.953Z cpu6:2054)NFS: 292: Restored connection to the server 10.13.111.197 mount point /tank/ISO, mounted as 5acdbb3e-410e56e3-0000-000000000000 ("ISO (1)")
2011-11-30T06:40:08.242Z cpu6:2054)NFSLock: 608: Stop accessing fd 0x41000a4c3e68  3
2011-11-30T06:40:34.647Z cpu3:2051)NFSLock: 568: Start accessing fd 0x41000a4d8928 again
2011-11-30T06:44:42.663Z cpu1:2058)WARNING: NFS: 283: Lost connection to the server 10.13.111.197 mount point /sastank/VMStorage, mounted as f0342e1c-19be66b5-0000-000000000000 ("SAStank")
2011-11-30T06:44:53.973Z cpu0:4011)NFS: 292: Restored connection to the server 10.13.111.197 mount point /sastank/VMStorage, mounted as f0342e1c-19be66b5-0000-000000000000 ("SAStank")
2011-11-30T06:51:28.296Z cpu5:2058)NFSLock: 608: Stop accessing fd 0x41000ae3c528  3
2011-11-30T06:51:44.024Z cpu4:2052)NFSLock: 568: Start accessing fd 0x41000ae3b8e8 again
2011-11-30T06:56:30.758Z cpu4:2058)WARNING: NFS: 283: Lost connection to the server 10.13.111.197 mount point /sastank/VMStorage, mounted as f0342e1c-19be66b5-0000-000000000000 ("SAStank")
2011-11-30T06:56:53.389Z cpu7:2055)NFS: 292: Restored connection to the server 10.13.111.197 mount point /sastank/VMStorage, mounted as f0342e1c-19be66b5-0000-000000000000 ("SAStank")
2011-11-30T07:01:50.350Z cpu6:2054)ScsiDeviceIO: 2316: Cmd(0x41240072bc80) 0x12, CmdSN 0x9803 to dev "naa.600508e000000000505c16815a36c50d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2011-11-30T07:03:48.449Z cpu3:2051)NFSLock: 608: Stop accessing fd 0x41000ae46b68  3
2011-11-30T07:03:57.318Z cpu4:4009)NFSLock: 568: Start accessing fd 0x41000ae48228 again

(I've put a complete dump from one of the hosts on pastebin: http://pastebin.com/Vn60wgTt)

When I got in the office at 9am, I saw various failures and alarms and troubleshooted the issue. It turned out that pretty much all of the VMs were inaccessible, and that the ESX hosts either were describing each VM as 'powered off', 'powered on', or 'unavailable'. The VMs described as 'powered on' where not in any way reachable or responding to pings, so this may be lies.

There's absolutely no indication on the X1600 that anything was awry, and nothing on the switches to indicate any loss of connectivity. I only managed to resolve the issue by rebooting the ESX hosts in turn.

I have a number of questions:

  1. What the hell happened?
  2. If this was a temporary NFS failure, why did it put the ESX hosts into a state from which a reboot was the only recovery?
  3. In the future, when the NFS server goes a little off-piste, what would be the best approach to add some resilience? I've been looking at budgeting for next year and potentially have budget to purchase another X1600/D2700/disks, would an identical mirrored disk setup help to mitigate these sorts of failures automatically?

Edit (Added requested details)

To expand with some details as requested:

The X1600 has 12x 1TB disks lumped together in mirrored pairs as tank, and the D2700 (connected with a mini SAS cable) has 12x 300GB 10k SAS disks lumped together in mirrored pairs as sastank

zpool status

  pool: rpool
 state: ONLINE
 scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        rpool       ONLINE       0     0     0
          c7t0d0s0  ONLINE       0     0     0

errors: No known data errors

  pool: sastank
 state: ONLINE
 scan: scrub repaired 0 in 74h21m with 0 errors on Wed Nov 30 02:51:58 2011
config:

        NAME         STATE     READ WRITE CKSUM
        sastank      ONLINE       0     0     0
          mirror-0   ONLINE       0     0     0
            c7t14d0  ONLINE       0     0     0
            c7t15d0  ONLINE       0     0     0
          mirror-1   ONLINE       0     0     0
            c7t16d0  ONLINE       0     0     0
            c7t17d0  ONLINE       0     0     0
          mirror-2   ONLINE       0     0     0
            c7t18d0  ONLINE       0     0     0
            c7t19d0  ONLINE       0     0     0
          mirror-3   ONLINE       0     0     0
            c7t20d0  ONLINE       0     0     0
            c7t21d0  ONLINE       0     0     0
          mirror-4   ONLINE       0     0     0
            c7t22d0  ONLINE       0     0     0
            c7t23d0  ONLINE       0     0     0
          mirror-5   ONLINE       0     0     0
            c7t24d0  ONLINE       0     0     0
            c7t25d0  ONLINE       0     0     0

errors: No known data errors

  pool: tank
 state: ONLINE
 scan: scrub repaired 0 in 17h28m with 0 errors on Mon Nov 28 17:58:19 2011
config:

        NAME         STATE     READ WRITE CKSUM
        tank         ONLINE       0     0     0
          mirror-0   ONLINE       0     0     0
            c7t1d0   ONLINE       0     0     0
            c7t2d0   ONLINE       0     0     0
          mirror-1   ONLINE       0     0     0
            c7t3d0   ONLINE       0     0     0
            c7t4d0   ONLINE       0     0     0
          mirror-2   ONLINE       0     0     0
            c7t5d0   ONLINE       0     0     0
            c7t6d0   ONLINE       0     0     0
          mirror-3   ONLINE       0     0     0
            c7t8d0   ONLINE       0     0     0
            c7t9d0   ONLINE       0     0     0
          mirror-4   ONLINE       0     0     0
            c7t10d0  ONLINE       0     0     0
            c7t11d0  ONLINE       0     0     0
          mirror-5   ONLINE       0     0     0
            c7t12d0  ONLINE       0     0     0
            c7t13d0  ONLINE       0     0     0

errors: No known data errors

The filesystem exposed over NFS for the primary datastore is sastank/VMStorage

zfs list

NAME                          USED  AVAIL  REFER  MOUNTPOINT
rpool                        45.1G  13.4G  92.5K  /rpool
rpool/ROOT                   2.28G  13.4G    31K  legacy
rpool/ROOT/solaris           2.28G  13.4G  2.19G  /
rpool/dump                   15.0G  13.4G  15.0G  -
rpool/export                 11.9G  13.4G    32K  /export
rpool/export/home            11.9G  13.4G    32K  /export/home
rpool/export/home/andrew     11.9G  13.4G  11.9G  /export/home/andrew
rpool/swap                   15.9G  29.2G   123M  -
sastank                      1.08T   536G    33K  /sastank
sastank/VMStorage            1.01T   536G  1.01T  /sastank/VMStorage
sastank/comstar              71.7G   536G    31K  /sastank/comstar
sastank/comstar/sql_tempdb   6.31G   536G  6.31G  -
sastank/comstar/sql_tx_data  65.4G   536G  65.4G  -
tank                         4.79T   578G    42K  /tank
tank/FTP                      269G   578G   269G  /tank/FTP
tank/ISO                     28.8G   578G  25.9G  /tank/ISO
tank/backupstage             2.64T   578G  2.49T  /tank/backupstage
tank/cifs                     301G   578G   297G  /tank/cifs
tank/comstar                 1.54T   578G    31K  /tank/comstar
tank/comstar/msdtc           1.07G   579G  32.8M  -
tank/comstar/quorum           577M   578G  47.9M  -
tank/comstar/sqldata         1.54T   886G   304G  -
tank/comstar/vsphere_lun     2.09G   580G  22.2M  -
tank/mcs-asset-repository    7.01M   578G  6.99M  /tank/mcs-asset-repository
tank/mscs-quorum               55K   578G    36K  /tank/mscs-quorum
tank/sccm                    16.1G   578G  12.8G  /tank/sccm

As for the networking, all connections between the X1600, the Blades and the switch are either LACP or Etherchannel bonded 2x 1Gbit links. Switch is a single Cisco 3750.

Storage traffic sits on its own VLAN segregated from VM machine traffic.

share|improve this question
    
Did you find a solution? –  ewwhite Mar 21 '12 at 12:41
    
I think the solution was to upgrade from Solaris 11 Express to Solaris 11. This seems to have fixed everything. –  growse Mar 21 '12 at 12:55
1  
(Get NexentaStor)! –  ewwhite Mar 21 '12 at 13:09
    
Can't. Corporate standards. Blah blahblahbalha :( –  growse Mar 21 '12 at 14:51
    
Its been quite a while since you had this problem. Did you ever figure out the solution? I did a Express 11 to 11/11 upgrade and it was worse. I have been rebooting on a schedule to avoid this but I would like to fix it. –  user157392 Feb 3 '13 at 20:58

1 Answer 1

I feel like we're missing the important information here. Please describe your storage server. The HP X1600 is basically a ProLiant DL180 G6. You're running that with an external HP D2700 enclosure connected via SAS cabling. I'm assuming this setup is running Solaris and has ZFS volumes shared out to your ESXi nodes. Some places to start are:

  • How are your zpools/filesystems configured?
  • Since both the X1600 and D2700 have disk bays, where are the NFS storage pools located (controller issue)?
  • Did you have corresponding errors on the Solaris system (zpool status -v output)?
  • How resilient is your networking for the NFS shares/traffic? Are you bonding, etc.?
  • Is your storage network isolated from LAN traffic (VLANs, separate switches)?
  • Did you tune your ESXi hosts using the NFS best-practices (heartbeat, timeouts, buffers)?
share|improve this answer
    
I've added some more details to the original post that hopefully covers your questions. –  growse Nov 30 '11 at 16:01
    
Oh, and are you using compression or dedupe? Based on what you have, I'd adjust the ESXi NFS parameters per the document linked in my answer. You may have had a small networking/resource issue on yoru storage server. I think it will require a reboot of the ESXi hosts. –  ewwhite Nov 30 '11 at 16:11
    
No compression or dedup - I've run into problems with them in the past. I've looked through the document and my NFS.HeartbeatFrequency, HeartbeatMaxFailures and HeartbeatTimout values are all as specified. I thought it was a server issue given that I saw the same type of failure on multiple clients at the same time... –  growse Nov 30 '11 at 16:48

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.