4

We had to perform a live volume resize of an NFS datastores VMWare uses on our Netapp. All of our Windows VMs were fine after the resize. However, some of our Linux VMs had issues.

Some Linux VMs just stopped responding. After restarting those VMs, I couldn't find anything in the logs indicative of an issue though.

I did find these kind of log messages on some of the VMs however:

May 29 14:56:02 rhel6-server-1314 kernel: INFO: task jbd2/dm-0-8:382 blocked for more than 120 seconds.
May 29 14:56:02 rhel6-server-1314 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 29 14:56:02 rhel6-server-1314 kernel: jbd2/dm-0-8   D 0000000000000000     0   382      2 0x00000000
May 29 14:56:02 rhel6-server-1314 kernel: ffff880037ce9c20 0000000000000046 ffff880037ce9be0 ffffffffa00041fc
May 29 14:56:02 rhel6-server-1314 kernel: ffff880037ce9b90 ffffffff81012b59 ffff880037ce9bd0 ffffffff8109b809
May 29 14:56:02 rhel6-server-1314 kernel: ffff880037ce1af8 ffff880037ce9fd8 000000000000f4e8 ffff880037ce1af8
May 29 14:56:02 rhel6-server-1314 kernel: Call Trace:
May 29 14:56:02 rhel6-server-1314 kernel: [<ffffffffa00041fc>] ? dm_table_unplug_all+0x5c/0x100 [dm_mod]
...             rhel6-server-1314
May 29 14:56:02 rhel6-server-1314 kernel: [<ffffffff8100c140>] ? child_rip+0x0/0x20
May 29 14:56:02 rhel6-server-1314 kernel: INFO: task master:1674 blocked for more than 120 seconds.
May 29 14:56:02 rhel6-server-1314 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 29 14:56:02 rhel6-server-1314 kernel: master        D 0000000000000000     0  1674      1 0x00000080
May 29 14:56:02 rhel6-server-1314 kernel: ffff88003d669958 0000000000000086 ffff88003d669918 ffffffffa00041fc
May 29 14:56:02 rhel6-server-1314 kernel: 0000000000000000 ffff880002216028 ffff880002215fc0 ffff88003fac2b78
May 29 14:56:02 rhel6-server-1314 kernel: ffff88003fac30f8 ffff88003d669fd8 000000000000f4e8 ffff88003fac30f8
May 29 14:56:02 rhel6-server-1314 kernel: Call Trace:
May 29 14:56:02 rhel6-server-1314 kernel: [<ffffffffa00041fc>] ? dm_table_unplug_all+0x5c/0x100 [dm_mod]
...             rhel6-server-1314
May 29 14:56:02 rhel6-server-1314 kernel: [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
May 29 14:56:02 rhel6-server-1314 kernel: INFO: task pickup:6197 blocked for more than 120 seconds.
May 29 14:56:02 rhel6-server-1314 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 29 14:56:02 rhel6-server-1314 kernel: pickup        D 0000000000000000     0  6197   1674 0x00000080
May 29 14:56:02 rhel6-server-1314 kernel: ffff88003da95968 0000000000000086 ffff88003da95928 ffffffffa00041fc
May 29 14:56:02 rhel6-server-1314 kernel: ffff88003da95938 ffff8800022128a0 ffff88003da95908 ffffffff81127ed0
May 29 14:56:02 rhel6-server-1314 kernel: ffff88003d90da78 ffff88003da95fd8 000000000000f4e8 ffff88003d90da78
May 29 14:56:02 rhel6-server-1314 kernel: Call Trace:
May 29 14:56:02 rhel6-server-1314 kernel: [<ffffffffa00041fc>] ? dm_table_unplug_all+0x5c/0x100 [dm_mod]
...             rhel6-server-1314
May 29 14:56:02 rhel6-server-1314 kernel: [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
May 29 14:56:02 rhel6-server-1314 kernel: mptscsih: ioc0: attempting task abort! (sc=ffff880037bfd280)
May 29 14:56:02 rhel6-server-1314 kernel: sd 2:0:0:0: [sda] CDB: Write(10): 2a 00 03 14 e8 d0 00 00 18 00
May 29 14:56:02 rhel6-server-1314 kernel: mptscsih: ioc0: WARNING - Issuing Reset from mptscsih_IssueTaskMgmt!! doorbell=0x24000000
May 29 14:56:02 rhel6-server-1314 kernel: mptscsih: ioc0: task abort: SUCCESS (rv=2002) (sc=ffff880037bfd280)
May 29 14:56:02 rhel6-server-1314 kernel: scsi target2:0:0: Beginning Domain Validation
May 29 14:56:02 rhel6-server-1314 kernel: scsi target2:0:0: Domain Validation skipping write tests
May 29 14:56:02 rhel6-server-1314 kernel: scsi target2:0:0: Ending Domain Validation
May 29 14:56:02 rhel6-server-1314 kernel: scsi target2:0:0: FAST-40 WIDE SCSI 80.0 MB/s ST (25 ns, offset 127)

My questions:

  1. Does anyone know what is causing this?
  2. If not, where else should we look for clues?
  3. Finally, does anyone know how to mitigate this next time we have to do a volume resize?
Improve this question
5
  • It might help to know what sizes "from" and "to" you resized the datastore. Also, logs from vSphere and/or the host(s) in question might help, too.
    – mfinni
    Commented Jun 24, 2014 at 13:14
  • Unfortunately, those vSphere logs are gone now. We added 1TB to the volume.
    – Belmin Fernandez
    Commented Jun 24, 2014 at 14:45
  • Are vmware tools installed on the VMs? Also check if you are affected by vmware KB2016122 or KB2076392
    – lacasitos
    Commented Jun 24, 2014 at 21:48
  • What type of storage is this? Is it a Netapp? (you should put that in the question body if it is)
    – ewwhite
    Commented Jun 26, 2014 at 13:14
  • VMWare tools is installed. Will check on the KB articles.
    – Belmin Fernandez
    Commented Jun 28, 2014 at 14:16

2 Answers 2

Reset to default
4
+50

I think, it was just an I/O timeout.

I had such issues with Linux VMs on remote NFS datastore. NFS was just too slow, and some of our Linux VMs switched their disks into read-only mode (and therefore stopped responding). Probably, during resize your NFS datastore was overloaded and this caused issues. Do Linux VMs work fine after reboot?

To avoid such issues, and slightly improve I/O performance for Linux guests, you could try to switch I/O scheduler to "noop" or "deadline" for all guests:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2011861

In my case, we had such timeout issues approximately once a week on the most loaded Linux guests, even with "scheduler fix". To solve it, we switched from NFS to iSCSI (you could also try to optimise your NFS settings like "rsize", "wsize", MTU and so on, but it was still not enough in my case) and tried to reduce I/O operation on guests whenever it was possible.

Improve this answer
2
  • After rebooting the affected Linux VMs, everything works fine. There's even been instances where we are able to set a r/o disk to r/w and everything is okay. We are trying to recreate a test to see if we could reproduce the issue and better troubleshoot it with your recommended steps.
    – Belmin Fernandez
    Commented Jun 30, 2014 at 16:02
  • From my experience, NFS is especially slow on small files. So for your test you could first try writing a lot of small files to the disk on one of the Linux guests, or just writing to several files in parallel line by line (at least 50 lines per second, and you should flush data to file after every line). However, be careful, as it could affect all other VMs using the same NFS datastore.
    – Andrey Sapegin
    Commented Jul 1, 2014 at 9:01
1

If this is a NetApp (or any other NFS server), be sure that the NFS best-practices for the ESXi host configurations are in place.

For NFS deployments, I always make some adjustments to NFS heartbeat and timeout settings. That may apply in your case. Check with your storage engineer to see what the specific recommendations for your unit are.

Improve this answer
2
  • Thanks. Going through the recommendations now. Would I not see issues on both Windows and Linux VMs if the problem lied in the ESXi's host NFS layer?
    – Belmin Fernandez
    Commented Jun 30, 2014 at 15:59
  • The individual operating systems react to this differently. I can tell you that I once had the NFS head of an EMC VNX5500 reboot spontaneously, and it forced the ext3 filesystems of 200+ Linux virtual machines in VMware to go read-only... but the Windows VMs were fine.
    – ewwhite
    Commented Jun 30, 2014 at 16:02

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .