Mirrored virtual disk on storage pool can't expand after adding drive on Windows Server 2012 R2

Question

I've recently taken to converting a couple of mirrored RAID1 arrays into one large storage pool. I have two 2TB disks and two 3TB disks, altogether 10TB, which when mirrored should give me 5TB of usable space.

I started the storage pool with the two 2TB disks and one of the two 3TB disks, using the final 3TB disk as a backup of the old data to move to the new pooled storage. I set up the virtual disk as "mirrored" on top of the new pool, and had a little under 3TB of usable space.

After moving the backup from the extra 3TB drive to the new pool, I cleaned it, and added it to the pool. However, I'm now unable to extend the mirrored virtual disk to take advantage of the added space.

Unfortunately I cannot post images due to my starter reputation, but my pool shows up with 9.09TB capacity, and 2.64TB of free space. When attempting to extend the virtual disk, the maximum size allowed is 3.22 TB, only just a hair more then before I added the 3TB drive. Physical disks show that almost none of the new disk is being used, while the rest are full.

I've read elsewhere that number of columns can restrict how you can extend virtual disks, but my number of columns is set to 1, which should allow extending to any number of disks.

Here's the powershell output of my virtual disk:

ObjectId                          : {1}\\SERVER\root/Microsoft/Windows/Storage/Providers_v2\SPACES_VirtualDisk.ObjectId
                                    ="{fdf741fe-cae8-11e4-80b4-806e6f6e6963}:VD:{d734cabd-cabf-11e4-80bf-000c41ebb9a3}{
                                    d734cad6-cabf-11e4-80bf-000c41ebb9a3}"
PassThroughClass                  :
PassThroughIds                    :
PassThroughNamespace              :
PassThroughServer                 :
UniqueId                          : D6CA34D7BFCAE41180BF000C41EBB9A3
Access                            : Read/Write
AllocatedSize                     : 3545495502848
DetachedReason                    : None
FootprintOnPool                   : 7090991005696
FriendlyName                      : McAfee Primary
HealthStatus                      : Healthy
Interleave                        : 262144
IsDeduplicationEnabled            : False
IsEnclosureAware                  : False
IsManualAttach                    : False
IsSnapshot                        : False
LogicalSectorSize                 : 4096
Name                              :
NameFormat                        :
NumberOfAvailableCopies           :
NumberOfColumns                   : 1
NumberOfDataCopies                : 2
OperationalStatus                 : OK
OtherOperationalStatusDescription :
OtherUsageDescription             :
ParityLayout                      : Unknown
PhysicalDiskRedundancy            : 1
PhysicalSectorSize                : 4096
ProvisioningType                  : Fixed
RequestNoSinglePointOfFailure     : False
ResiliencySettingName             : Mirror
Size                              : 3545495502848
UniqueIdFormat                    : Vendor Specific
UniqueIdFormatDescription         :
Usage                             : Other
WriteCacheSize                    : 0
PSComputerName                    :

And here is the powershell output for the pool:

PS H:\> Get-StoragePool -FriendlyName "McAfee Primary Pool"

FriendlyName            OperationalStatus       HealthStatus            IsPrimordial            IsReadOnly
------------            -----------------       ------------            ------------            ----------
McAfee Primary Pool     OK                      Healthy                 False                   False


PS H:\> Get-StoragePool -FriendlyName "McAfee Primary Pool" | FL


ObjectId                          : {1}\\SERVER\root/Microsoft/Windows/Storage/Providers_v2\SPACES_StoragePool.ObjectId
                                    ="{fdf741fe-cae8-11e4-80b4-806e6f6e6963}:SP:{d734cabd-cabf-11e4-80bf-000c41ebb9a3}"
PassThroughClass                  :
PassThroughIds                    :
PassThroughNamespace              :
PassThroughServer                 :
UniqueId                          : {d734cabd-cabf-11e4-80bf-000c41ebb9a3}
AllocatedSize                     : 7092601618432
ClearOnDeallocate                 : False
EnclosureAwareDefault             : False
FriendlyName                      : McAfee Primary Pool
HealthStatus                      : Healthy
IsClustered                       : False
IsPowerProtected                  : False
IsPrimordial                      : False
IsReadOnly                        : False
LogicalSectorSize                 : 4096
Name                              :
OperationalStatus                 : OK
OtherOperationalStatusDescription :
OtherUsageDescription             :
PhysicalSectorSize                : 4096
ProvisioningTypeDefault           : Fixed
ReadOnlyReason                    : None
RepairPolicy                      : Parallel
ResiliencySettingNameDefault      : Mirror
RetireMissingPhysicalDisks        : Auto
Size                              : 9998683865088
SupportedProvisioningTypes        : {Thin, Fixed}
SupportsDeduplication             : False
ThinProvisioningAlertThresholds   : {70}
Usage                             : Other
Version                           : Windows Server 2012 R2
WriteCacheSizeDefault             : Auto
WriteCacheSizeMax                 : 107374182400
WriteCacheSizeMin                 : 0
PSComputerName                    :
FileSystem                        : Unknown

Any ideas how I can reclaim my space?

Answer 1

I ran into the same issue. Neograph's comment is irrelevant in this case, he's talking about traditional Windows Server disk mirroring, not Storage Spaces. Thin provisioning is also quite irrelevant in this case. You can use it as a workaround, but I think more cautious planning will be better both budget- and performance-wise, just read on and you'll see.

After quite some time spent on reading and playing around in Server Manager, I think I figured out what's going on. The thing is, SS has this thing called "columns". That defines how many disks data is striped across. If your virtual disk was created with 4 columns, data is only spread across 4 disks with Simple layout (i.e. RAID0) or 8 disks with Two-way mirror (i.e. RAID10), not all of them. Now this may be confusing for someone who comes from traditional HW RAID (like me), but that's the way it is.

Note: from now on, I'll refer to the number of columns as column size because it's much more intuitive for me this way.

So anyway, column size also defines how you can extend a virtual disk. Clearly, if your current VD has groups of 4 disks (= column size is 4), you can't add a "half" group by adding 2 new disks. So the number of disks required for expansion is basically

n x NumberOfColumns x NumberOfDataCopies

So if you have a two-way mirror and a column size of 1, you can only add pairs of disks. If your mirror has a column size of 3, you can only add 6, 12, 18 or so disks.

From what I understand, the default column size for a VD is the number of disks divided by copy count, but 8 at maximum, e.g. if you have 10 disks in a two-way mirror, column size will be 5, if you have 16 disks, column size will be 8, but if you have 24 disks, column size will still be 8 - by default. Note: you can check these numbers under VD properties (NumberOfColumns and NumberOfDataCopies properties undes Details).

And here come a lot of headaches:

• the default column size requires you to double the disk count if you want to expand the VD (in most cases)
• you can only select column size if disk usage was set to Manual during pool creation
• the default disk usage is Automatic (of course)
• you cannot change column size once the VD's created
• you cannot change disk usage to Manual once the pool's created

So to have this set up properly, you need to delete:

• the volume
• the virtual disk
• the storage pool

i.e. everything. As a sidenote, stripe size (called Interleave size in SS) is also unavailable if disk usage is set to automatic.

Now you may wonder why would anyone use anything bigger than 1 for column size. The answer is of course performance. The bigger the column size the better the performance you get. Actually, it can be quite dramatic, here's a benchmark with column size 1 and 6:

You need to plan wisely. Only use a high column size if you know for sure that you'll be able to afford to purchase a big number of disks once disk space runs out.

Some good reads on the topic:

• Step By Step: How To Extend and Resize A Two-Way Mirrored Storage Tiered Space?
• Storage Spaces - Designing for Performance

Answer 2

I can't downvote by now, but I want to outline, that the basic information given in bviktors post is wrong - he is still thinking in Raid by saying you can't extend with half a diskgroup:

• If you have a Storage Pool with 4 Columns - you can use any number of disks, starting with 4.
• Storage Spaces will always utilize all the Disks you have!
• The Column count just defines on how much Disks data is written at the same time (This is called striping)
• The next Stripe (by default 256 KB in Size, called Interleave) however can (and will) be written to 4 different Disks!

Hence, to extend a Storage Pool of 4 Columns - you don't need to add 4 disks - you only need to make sure, that you have 4 disks with remaining diskspace left in the pool!

So, if 1 is full, but 3 have remaining space, the pool will become operational again after adding One Disk!

(This allows to mix capacity as desired - there's no need to keep the raid-constraint of equal disk-sizes)

As a Best-Practice, you should always add more disks, than #NumberOfDataCopies * #NumberofColumns would be:

Consider a 2 Column 2 Copy Disk - It requires a minimum of 4 disks. If you loose one Disk, you could still access your data - but you cannot write anything anymore, cause you don't have 4 columns left where data could be stored!

Consider you would have added 5 Disks to that pool (which will be used based on Size by the Storage Spaces Subsystem, filled up in the best possible way to make all disks hit 100% at the same time) - loosing one Disk still retains your data - and keeps your Pool working for new writes, because you still have the minimum of 4 Columns left.

Also, this allows you to rebuild the pool immediately if one disk fails, without having to purchase a new disk first!

 Set-PhysicalDisk -FriendlyName "BrokenDisk" -Usage Retired
 Get-PhysicalDisk -FriendlyName "BrokenDisk" | Get-VirtualDisk | Repair-VirtualDisk -AsJob

The data will now be "moved" to the remaining disks, if enough space is left. After the rebuild:

 $disk = Get-PhysicalDisk -FriendlyName "BrokenDisk"
 Remove-PhysicalDisk -StoragePoolFriendlyName "My Pool" -PhysicalDisks $disk

(You can use the same commands to retire "functional" disks and move data to other disks - this will allow some sort of redistributing the data, once you add a disk - but at the end you will always have one disk "empty". However in your case it would not work, due to the small number of disks. In 10 disk pool for instance, you could free up a 2 TB Disk, by distributing as little as 200 MB to every other disk. Re-Running the operation will now write prefered to the empty disk. Storage Spaces basically always says: "I have to write: 8 Blocks (NumberOfColumns * NumberOfDataCopies) with a size of 64 KB (Interleave / Number of Columns) each - give me 8 distinct disks out of the 10 disks with the least percentual usage, so I can throw the data there!")

---

In your example mentioned:

I started the storage pool with the two 2TB disks and one of the two 3TB disks, using the final 3TB disk as a backup of the old data to move to the new pooled storage. I set up the virtual disk as "mirrored" on top of the new pool, and had a little under 3TB of usable space.

You started with 2+2+3 TB - Storage Spaces now would put about 1/3 more data on the 3TB disk, than on the Other Disks while maintaining the required mirror, so they fill up equaly.

("2 Datacopies, 1 Column: I have to write: 2 Blocks (NumberOfColumns * NumberOfDataCopies) with a size of 256 KB (Interleave / Number of Columns) each - give me 2 distinct disks out of the 3 disks with the least percentual usage, so I can throw the data there!")

Physical disks show that almost none of the new disk is being used, while the rest are full.

Now, your disks are "almost full" and you added a "new" disk:

disk | size | usage
1    | 2TB  | 1.9 TB
2    | 2TB  | 1.9 TB
3    | 3TB  | 2.8 TB
4    | 3TB  | 0 TB

Now, remember you got a mirrored virtual disk there: If you are going to extend that disk, Storage Spaces needs two have at least 1MB on two Disks, to extend the virtual disk by 1MB

Your Pool now will do the following: EVERY Data written goes to disk 4 - and every COPY written goes to either 1,2 or 3, as this will give you the most usable Diskspace.

While Extending a (fixed) this, the following equation would be evaluated:

MAX_EXTEND = Math.MIN(100+100+200 , 3000) = 400 MB

A thin provisioned can be extended beyond the available physical size - it will just stop accepting data, once 400 MB have been written!

Hence, with the free diskspace of 100+100+200 - Your Virtual Disk can only be increased by 400MB, because then disk 1,2,3 are full - and disk 4 has 2.6 TB Space left. No more storage to keep a dual mirror operational.

Now, you just need to add 1 Disk - let's say 8 TB to be able to extend your virtual disk by 2.6 TB again (2.6 on disk 4 and 2.6 on the "new 8 TB Disk")