Removing VMFS Datastores from ESXi5.5 with VSAN

ESXi 5.5 and above stores coredumps on a datastore attached to the host, it can also create a vsantraces directory. Both of these can lock the datastore and prevent it from being deleted.

To check for and remove the coredump file do the following:

esxcli system coredump file list Path                               
Active  Configured       Size
-------------------------------------  ------  ----------  ---------
/vmfs/volumes/xx/vmkdump/xxx.dumpfile   false       false  702545920
/vmfs/volumes/xxx/vmkdump/xxx.dumpfile    true        true  702545920
/vmfs/volumes/xxx/vmkdump/xxx.dumpfile   false       false  702545920

The output shows that there are 3 dump files which are blocking the datastore. Only the owning ESXi host can disable and delete them, so you have to find out which ESXi is responsible for each file:

~ # vmkfstools -D /vmfs/volumes/4fcf1090-bfc93298-896d-f4ceae3333d/vmkdump/39333335-3431-5A33-3332-313333385042-733333920.dumpfile
 Lock [type 10c00001 offset 86437888 v 25, hb offset 3793408
 gen 1, mode 1, owner 5533361-0933347c-5b33-f43333393294 mtime 157
 num 0 gblnum 0 gblgen 0 gblbrk 0]
 Addr <4, 149, 6>, gen 8, links 1, type reg, flags 0, uid 0, gid 0, mode 600
 len 702545920, nb 84 tbz 0, cow 0, newSinceEpoch 84, zla 1, bs 8388608
 ~ #

You can see the UUID (bold) from the ESXi host that has locked the file. (The last part of the UUID is determined by the MAC address of vmnic0. f43333393294 = f4:33:33:39:32:94).

You can find the host by searching for this MAC using the following PowerShell:

PS C:\Users\jc; Get-VMHostNetworkAdapter |? {$_.Mac -eq "f4:33:33:39:32:94"} | select VMhost, Name, Mac
 VMHost                  Name        Mac
 ------                  ----        ---
 esxihost.mydomain.com   vmnic8      f4:33:33:39:32:94

To remove the coredump file, connect to the ESXi host with SSH and use the esxli system coredump file remove command. This will remove the configured active coredump file:

~ # esxcli system coredump file remove --force

To check for and remove the vsantraced files do the following:

~ # lsof |grep vsantraced |grep volumes
 34412 vsantraced FILE 3 /vmfs/volumes/4f333330-bfc33298-893d-f4ce46a9393d/vsantraces/vsantraces-- 2015-06-03T16h11m36s899.gz

To remove the log stop vsantraced, unmount/delete the datastore and start the service again:

~ # /etc/init.d/vsantraced stop
 Remove the Datastore
 ~ # /etc/init.d/vsantraced start

Leave a Reply

Your email address will not be published. Required fields are marked *