Cannot open the disk and could not open change tracking file

Lately I have seen a number of VMs going down after the backup has finished and snapshots get removed. At some point when cleaning the snapshot, the ESXi host decides something is wrong and then brings down the VM. After this we’re unable to power on the VM and receive the error: “Cannot open the disk ‘/vmfs/volumes/xxxxxxxxxx/xxxxx.vmdk’ or one of the snapshot disks it depends on”. When you then check the event details of the VM, you’ll see two messages:

– “Cannot open the disk ‘/vmfs/volumes/xxxxxxxxxx/xxxxx.vmdk’ or one of the snapshot disks it depends on”.
– “Could not open/create change tracking file”.

 

cbt001

The fix to make the VM power on again is simple, however we still haven’t been able to solve the root cause. VMware Support has suggested to bring down the number of LUNs connected to a single host and to upgrade to vSphere 5.5.

The fix:

– Make sure the VM is down
– Go to Edit Settings -> Options -> Advanced / General -> Press the “Configuration Parameters” button.
– Search the list for the name: “ctkEnabled” and change the value to “false”.
– Search the list for the names: “scsi0:0.ctkEnabled”, “scsi0:1.ctkEnabled”, “scsi1:0.ctkEnabled”, etc…. and set these values to “false”.
– Click OK a few times.

cbt002

Last step is to remove the reference to the change tracking file from the vmdk descriptors.

– Enable SSH on you ESXi host and login through SSH
– Go to the directory of the VM that holds the VMDK file. If your VM has multiple VMDKs, maybe spread over multiple datastores, you’ll have to repeat this for each VMDK.
– List all VMDK files:  ls -l *.vmdk
– Check which VMDK file still has a reference to the change tracking files (CBT): grep changeTrackPath VMName.vmdk
– You should see something like this: changeTrackPath=”SPVSQ001-ctk.vmdk”
– If the references is still present, edit the vmdk file using the vi editor and place a # at the start of the changeTrackPath line. (Go to the line, press i for insert, type #, press <ESC>:wq to save the VMDK).
– Check the other VMDKs as well but leave the “-flat” and “-ctk” vmdk files alone.
– Now try to Power On the VM.

cbt004

When the VM is running, don’t forget to exit your SSH session and disable SSH Service on the host.