More VSAN Components than Physical Disks

When running through some VSAN operational readiness tests I stumbled across an issue when simulating host failures. When there are more VSAN Components than physical disks and a host fails, the components will not be rebuilt on remaining hosts.

Firstly here is some background information about the test cluster:

  • 4 x Dell R730XD Servers
  • 1 Disk Group per server with one 800GB SSD fronting six 4TB Magnetic Disks
  • 1 Test VM with a single 1.98TB VMDK
  • Disks to Stripe set to 1 on the storage policy applied to the VM
  • Failure to Tolerate set to 1 on the storage policy applied to the VM
  • ESXi 5.5 and VSAN 5.
  • All drivers/firmware on the VSAN HCL

The VMDK Object is split into 24 components (8 x “Primary” components (each 250GB), 8 x “Copy” components (each 250GB) and 8 x “Witness” components

Note: VSAN does not really have “Primary” and “Copy” components but for the sake of the following diagrams and ease of explanation I’ll group the components this way.

As below:

More VSAN Components than Physical Disks - Failure Scenario

More VSAN Components than Physical Disks – Failure Scenario

When this VM is deployed VSAN will stripe the “primary” and “copy” components so that no two components are placed on the same physical spinning disk. It will also attempt to place all components, if possible, on then same Disk Group. If this is not possible VSAN will attempt to place the components on different Disk Groups on the same ESXi Host. Finally if this is not possible it will place the components on individual physical disks on different hosts.

Below is a representation of the component placement (in this example VSAN cannot place all “Primary” or “Copy” components on the same Disk Group/ESXi Host). VSAN also cannot place “Primary” AND “Copy” components onto the same host. If it did this there is a risk that a host failure could remove access to RAID 0 components  from both RAID 1 groups.

More VSAN Components than Physical Disks - Failure Scenario

More VSAN Components than Physical Disks – Failure Scenario

With this configuration the VM Storage Policy will show all components as Active and the object as compliant.

If you fail any node in the cluster (pull the power), either 6 “Primary” or “Copy” components (in the event node A or C fail) or 2 “Primary” / “Copy ” and 4 “Witness” components (in the event node B or D fail) will have to be rebuilt onto other hosts.

More VSAN Components than Physical Disks - Failure Scenario

More VSAN Components than Physical Disks – Failure Scenario

In this example:

  • P1  – Cannot be rebuilt on ESXi Host B as W1 is already on this Host.
  • P1 – Cannot be rebuilt on ESXi Host C as C1 is already on this Host.
  • P1 – Cannot be rebuilt on ESXi Host D as there are “Copy” components on this Host. “Primary” and “Copy” RAID Zero components cannot exist on the same host.
  • P2 – Will be rebuilt onto ESXi Host B
  • P3  – Cannot be rebuilt on ESXi Host B as W3 is already on this Host.
  • P3 – Cannot be rebuilt on ESXi Host C as C3 is already on this Host.
  • P3 – Cannot be rebuilt on ESXi Host D as there are “Copy” components on this Host. “Primary” and “Copy” RAID Zero components cannot exist on the same host.
  • P4 – Will be rebuilt onto ESXi Host B
  • P5  – Cannot be rebuilt on ESXi Host B as W5 is already on this Host.
  • P5 – Cannot be rebuilt on ESXi Host C as C5 is already on this Host.
  • P5 – Cannot be rebuilt on ESXi Host D as there are “Copy” components on this Host. “Primary” and “Copy” RAID Zero components cannot exist on the same host.
  • P6 – Will be rebuilt onto ESXi Host B

Solutions

There are three possible solutions to this problem:

  • Limit the VMDK object size to:
    • 250GB if there is 1 Physical Disk per Disk Group
    • 500GB if there is 2 Physical Disks per Disk Group
    • 750GB if there is 3 Physical Disks per Disk Group
    • 1TB if there is 4 Physical Disks per Disk Group
    • 1.25TB if there is 5 Physical Disks per Disk Group
    • 1.5TB if there is 6 Physical Disks per Disk Group
    • 1.75TB if there is 7 Physical Disks per Disk Group
  • Increase the minimum number of nodes in a VSAN cluster to:
    • 4 to tolerate 1 host failure when each host has 8 physical disks
    • 5 to tolerate 1 host failure when each host has 6 physical disks
    • 7 to tolerate 1 host failure when each host has 4 physical disks
    • 9 to tolerate 1 host failure when each host has 3 physical disks
    • 13 to tolerate 1 host failure when each host has 2 physical disks
    • 25 to tolerate 1 host failure when each host has 1 physical disk
  • Increase the number of physical disks in each host to 8

 

Leave a Reply

Your email address will not be published. Required fields are marked *