1

We are running four ESXi 6.5.0 VMware hosts, managed vie vSphere web client.

Recently, I added a new VMFS 6 volume in our storage, originally created with 2TB capacity, but quickly extended to 3TB before actually taking it to production. Ever since, I do not get consistent volume cacpacity readings. For example: A guest system had a disk of 1700GB originally on a different volume; I migrated that to the new volume, and after that try to enlarge it. So in change settings, the disk showed as 1700GB and maximum size 2.77TB - but when I attempted to set it to 2000GB, I got an alarm "insufficient disk capacity", and - lo and behold - the maximum size had gone down to 1.86TB! With this, I managed to increase my 1700 to 1900GB only ...

When checking the volume info under "configure" - "general" - "capacity", it sometimes shows 1.86TB as capacity (and allocated), but sometimes without anything special happening in-between it shows 2.77TB (with 1.86TB allocated). At times when the lower capacity is shown, clicking "update capacity" changes nothing.

Under "device backing", it shows 2.77TB (as far as I can tell without fluctuation?).

Under "monitor" - "performance" - "overview", I sometimes see 2.77 and sometime 1.86TB as well.

And so on ...

Each of the four hosts shows the volume correctly as LUN 10 = 2.77TB.

Strangely, the event log for the volume shows several entries "Capacity of ... enlarged from 2047894093824 bytes to 3047816167424 bytes". More specifically, I have the "volume created" event at 2019-07-05 11:44:16 and those "capacity extended" events at 2019-07-05 13:33:44, 2019-07-05 13:37:04, 2019-07-05 15:23:28, 2019-07-05 15:53:40, 2019-07-05 16:23:41, 2019-07-05 17:19:30, 2019-07-10 07:58:05, 2019-07-10 09:58:01. That's eight times up to now, way more than e.g. once per host, and mostly (but not completely) correlated with times when I have vsphere open, all in all not really explicable to me.

Questions:

  • What can be the cause of these observed fluctuations?
  • How can it be mended?
  • Once mended, how can I then be sure that it has been mended for good (I do not want to extend a disk an then be informed at a random moment that pat of the disk does not "exist")?
1

I have a few ideas, I can't confirm whether they're actually what you're seeing though:

  1. What version of VMFS are you running on each system? Older versions appear to handle large files (near 2tb) differently than newer ones.
  2. Are there any volume healing operations you can try? It could be that the metadata on one or more of the disks is out of date and reports the wrong values when VMware tries to read it.
  3. How big are the individual storages used as VMFS backend? Are they large enough to hold a single 2TB file? I'm not sure if VMFS can distribute single VM disks across multiple nodes.
  4. I actually don't know VMFS that well, but I do know that ZFS and BTRFS (which are Copy on Write filesystems) are sometimes unable to precisely tell you the size of individual files on disk due to cluster/inode slack, snapshots, metadata, and other things that may not be counted when just querying the file size itself. With ZFS you can do zfs list -o space to show exactly what's used for what, perhaps VMFS has some similar option?

Here's the wikipedia page of VMFS for reference, note that it has a bunch of constraints especially in older versions that may be relevant to your use case. https://en.wikipedia.org/wiki/VMware_VMFS

Hope this helps!

| improve this answer | |
0

Could your server be swapping memory into disk? Vmware has an option to disable this: https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.resmgmt.doc/GUID-C0DBB2A3-4B44-45BE-927E-10EEB2EB8CBE.html

| improve this answer | |
  • Which server? The guest system, which has only 32GB as RAM? I'd be surprised if that lead to 1TB fluctuations on a newly created volume which happens to have been enlarged by 1TB after creation, whereas no such behaviour was observed while the guest disk resided on an old volume ... – Hagen von Eitzen Sep 8 '19 at 7:11
-1

There is no variation.

It depends how space is interpreted: as fake space in base 10 or as true space in base 2. HDD manufactures list the space in fake-space, in base 10 and in a similar manner some programs read it.

What that means for you: your fake-3TB volume actually has 3 000 000 000 000 B, which in true binary space means 2.72TB. If you want real 3TB, you will have to make a volume in size of 3 221 225 472 000 B.

In a similar manner, a 2TB base 10 capacity will actually have 1.81TB.

Your volume of 2047894093824 B is under 2TB. That would best translate as 2 000 000 000 KB which is 1.86TB. If you want 2TB, make it have 2199023255552 B (or 2200000000000 if it's easier for you).

Your allocation of 1700GB in TB has the size of 1.66TB. 1900G is 1.8555TB so that is why it worked.

To eliminate the confusion it was officially proposed a while ago that the binary true values to be stated in KiB/MiB/GiB/TiB and the 10-base ones with the old names, but many did not agree with this implementation of naming.

Therefore, if you create volumes make sure you take into account the binary values, not the 10-base translation of them.

If you create a 2048GB volume it will actually have exactly 2TB if the GB value you state is in binary format and will still have less if it's not.

| improve this answer | |
  • You missed the point. Base 10 vs base 2 could show the same volume e.g. sometimes as 3 TB and sometimes as 3.22 TB - with an "error" between the variants below 10 percent. But that cannot explain a display of sometimes ~3TB and sometimes ~2TB – Hagen von Eitzen Sep 8 '19 at 7:05
  • Yes you can and it's a little related to the same thing. 1.86TB means 2FakeTB and it means a software of reading function of an external call is limited to reading values below 2TB. In the case of exceeding values (like between 2 and 4TB) you may even see the real value minus 2TB. In have encountered such miss-reading in numerous older software that was not designed to handle very large data values. – Overmind Sep 9 '19 at 5:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.