Objective 1.2 – Manage Complex Storage Solutions

Analyze I/O workloads to determine storage performance requirements

Do this this we can use vscsiStats:

  • vscsiStats -l returns the “worldGroupID” of VMs running on the host.
  • Run vcsiStats -s -w [worldGroupID] to start the capture session.
  • To print the output and specify the type of histogram (all, ioLength, seekDistatnce, outstandingIOs, latency or interarrival) enter vscsiStats –p [histogram type], for example vscsiStats –p latency.

vscsistats

  • To redirect the capture to a file, enter vscsiStats –p latency > vm01.txt
  • To stop the collection, enter vscsiStats –x
  • To reset the stats, enter vscsiStats –r

Alternatively we can use esxtop/resxtop:

esxtop/resxtop can also be used to gather valuable I/O latency statistics. Applicable statistics include device latency, time spent in the kernel and latency seen by the guest operating system.

Disk throughput can be monitored using a few different esxtop metrics:

  • READ/s  – Number of disk reads per second.
  • WRITE/s – Number of disk writes per second.
  • MBREAD/s – Number of megabytes read per second.
  • MBWRTN/s – Number of megabytes written per second.

The sum of reads and writes equals IOPS, the most common benchmark when monitoring and troubleshooting storage performance. Using resxop, these metrics can be monitored at the HBA or VM level, providing flexibility in performance troubleshooting isolation.

Additional metrics are available in resxtop to assist an administrator in analyzing I/O workloads, specifically latency.

  • DAVG/cmd – The average amount of time it takes the physical storage system (from the HBA to the array) to process a single read or write I/O request. A value exceeding 20 warrants a closer look.
  • KAVG/cmd – The average amount of time the vmkernel (CPU) takes to process a SCSI operation. The KAVG value should be near zero considering the speed of modern day CPUs. Any value exceeding 0 might indicate excessive latency.
  • GAVG/cmd  – Represents the sum of DAVG + KAVG, or the total latency experienced by the guest operating system.

esxtop

Identify and tag SSD and local devices

  • Login to your ESXi Host
  • Type: esxcli storage nmp device list
  • This command provides a full listing of available devices:

MarkSSD

  •  You can add a claim rule by specifying the device name:
    • esxcli storage nmp satp rule add -s SATP  –device device_name –option=”enable_ssd”
  • You can add a claim rule by specifying the vendor name and the model name.
    • esxcli storage nmp satp rule add -s SATP  -V vendor_name -M model_name –option=”enable_ssd”
  • You can add a claim rule based on the transport protocol.
    • esxcli storage nmp satp rule add -s SATP  –transport transport_protocol –option=”enable_ssd”
  • You can add a claim rule based on the driver name.
    • esxcli storage nmp satp rule add -s SATP  –driver driver_name –option=”enable_ssd”
  • Reclaim the device.
    • esxcli storage core claiming reclaim –device device_name
  • Verify if devices are tagged as SSD.
    • esxcli storage core device list -d device_name

Administer hardware acceleration for VAAI

  • Connect to vCenter Server
  • Navigate to the Hosts and Clusters view
  • Select the host
  • Click Configuration > Hardware > Storage
  • Ensure that the Datastores view is selected
  • Note the status of the Hardware Acceleration column. (Status will be either Supported, Not Supported or Unknown.)

VAAIVAAI2

This can also be done on the command line:

esxcli storage core device vaai status get

VAAI3

In the event that VAAI needs to be disabled, potentially due to an array compatibility issue, perform the following steps:

  • Connect to vCenter Server
  • Navigate to Hosts and Clusters view
  • Click Configuration > Software > Advanced Settings
  • Change the values for three advanced settings to zero
    • DataMover.HardwareAcceleratedInit
    • DataMover.HardwareAcceleratedMove
    • VMFS3.HardwareAcceleratedLocking

Configure and administer profile-based storage

Create Storage Capabilities and Profiles

  • Enter the VM Storage Profiles view
  • Click Enable VM Storage Profiles
  • Click Manage Storage Capabilities
  • You may see a list of System-Defined capabilities. These are provided by your Storage API – Storage Awareness enabled storage. Additionally, you may add your own user-defined capability by clicking Add.
  • Provide a name and description for your user-defined storage capability and click OK.
  • When you are finished adding new user-defined capabilities, you may exit the Manage Storage Capabilities dialog by clicking Close
  • Click Create a VM Storage Profile
  • Enter both a name and description for your new storage profile and then click Next
  • Associate a Storage Profile with a storage capability and then click Next. Note, you may add more than one capability to a storage profile.

Attach Storage Capability to Datastores/Datastore Clusters

A user-defined storage capability may not be applied to a datastore cluster. However if all datastore cluster members have the same storage capability, then the datastore cluster will implicitly have the same storage capability.

  • Enter the Host and Clusters view
  • Select the appropriate host and then click the configuration tab
  • Select storage from within the Hardware frame
  • Ensure that the datastore view is selected
  • Right click the appropriate datastore or optional select more than one datastore to assign a storage capability to more than one datastore at the same time
  • Select Assign User-defined storage capability
  • For the Name, select the appropriate capability
  • Optionally, click New to create a new user-defined capability
  • Click OK

Attach Storage Profiles to a Virtual Machine

  • Enter the Host and Clusters view
  • Right-click the virtual machine to attach a storage profile
  • Select VM Storage Profile > Manage Profiles
  • On the profiles tab, under Home VM Storage Profile, select the appropriate profile
  • You may now click Propagate to Disk to apply the same Storage Profile to all virtual disks or optionally individually select a Storage Profile for each virtual disk attached to the VM
  • Click OK

Check Storage Profile Compliance for a Virtual Machine

  • Select VM Storage Profile > Check Profiles Compliance
  • Right-click the virtual machine to attach a storage profile
  • To force a Storage Profile Compliance check, Enter the Host and Clusters view
  • To check compliance, select a virtual machine
  • Click the summary tab
  • Under VM Storage Profiles view Profiles Compliance

Prepare storage for maintenance

Before you unmount a datastore, consider:

  • No virtual machines are powered on that are located on the datastore
  • If you have plans to destroy or modify the storage, that all virtual machines and data are relocated to another datastore
  • The datastore is not used for HA heartbeating
  • Storage I/O Control is disabled for the datastore
  • If a member, remove the datastore from a datastore cluster and unaffiliated the datastore from SDRS
  • Verify that the datastore is not associated with Host Cache

To Unmount a LUN

  • Login to the vSphere client as an administrator
  • Enter the Host view
  • Select the Host with the attached datastore
  • Click the Configuration tab
  • Click on Storage within the Hardware frame
  • Locate the Datastore to unmount
  • Right click the datastore and select Properties
  • Uncheck Enabled under Storage I/O Control and then click Close
  • Right click the datastore and select Enter SDRS Maintenance Mode
  • If necessary, right click and browse the datastore to ensure that all data was migrated. If not, vMotion or manually move the content to another datastore
  • Enter the Datastore view
  • Remove the Datastore from the cluster by dragging the storage to Local Storage
  • Right Click the Datastore and select Unmount. You should be greeted by this screenshot:

To mount a LUN

  • Enter the Datastore view
  • Locate and right click the inactive Datastore and select Mount
  • Right click the Datastore and select Exit SDRS Maintenance mode
  • If required, reactivate the applicable storage features: Storage I/O Control, Host Cache, HA

Apply space utilization data to manage storage resources

I suppose VMware are looking for decisions based on the output from vscsiStats and esxtop… not sure but I suspect I may have to balance capacity or IOPs across datastores.

Provision and manage storage resources according to Virtual Machine requirements

To create an empty virtual disk using vmkfstools:

  • the -c parameter specifies you want to create a new virtual disk followed by the required size.
  • the -d parameter specifies the disk format: zeroedthick, thin and eagerzeroedthick
  • the -a specifies the adapter type; buslogic, lsilogic, ide (default is buslogic)

Here we will create a 300 GB thin disk named temp.vmdk with the default buslogic adapter in a virtual machine folder named JC-Temp-VM:

vmkfstools -c 30G -d thin ‘/vmfs/volumes/LUN3/JC-Temp-VM/temp.vmdk

Understand interactions between virtual storage provisioning and physical storage provisioning

 

Configure datastore Alarms

 

Create and analyze datastore alarms and errors to determine space availability