Verifying ESX virtual machine file integrity.

Symptoms

• Unable to power on the virtual machine
• Powering on the virtual machine fails
• You see one of these errors:
  • A file was not found.
  • Operation not allowed in the current state
• The <vmname>.vmdk file does not exist, but the <vmname-flat>.vmdk file exists.
• Deleting snapshot fails with the error:
  
  ASSERT /build/mts/release/bora-39823/bora/vmx/snapshot/snapshot_vmx.c:752
• VMFS volume becomes inaccessible due to file system corruption

Purpose

This article describes the various files that make up a virtual machine running on an ESX host. It also provides information on how the integrity of these files may be compromised and the various means of resolving the resulting problems.

Resolution

Before ensuring virtual machine file integrity, be aware of the various files that make up a virtual machine running on an ESX Server host.

VMX File

The VMX file is the primary configuration file a virtual machine. Every aspect of your virtual machine is detailed in the VMX file.

Any virtual hardware assigned to your virtual machine is present here. An example entry for a virtual floppy drive is outlined below:

floppy0.startConnected = "false"
floppy0.clientDevice = "true" 

Each time you create a new virtual machine using the New Virtual Machine Wizard, the VMX file is appended to with each question you answer regarding the guest operating system, disk sizes, and networking.

VMXF File

This is a supplemental configuration file for virtual machines.

VMDK and -flat.VMDK Files

The <vm_name>-flat.vmdk files store the content of the virtual machines actual hard disk drives. Earlier versions of ESX used the extension .dsk for the virtual disk files.

The <vm_name>.vmdk files are the disk descriptor files. This file contains all the information about the associated flat file.

The files tell you the CID of the file, the parent CID of the file, and the type and size of the file amongst other things.

This information can be very useful depending on what type of issue you are troubleshooting. It is especially useful when troubleshooting problems with snapshots.

VMSD File

This is a centralized file for storing information and metadata about a snapshots of a virtual machine.

NVRAM File

This file contains the BIOS of the virtual machine.

Log Files

The vmware(-x).log files keep a log of key virtual machine activity. This file can be useful in troubleshooting if you encounter problems.

---

Renaming a virtual machine

Issues can often arise from people trying to rename their virtual machine, causing inconsistencies.

The proper procedure to rename a virtual machine is:

Note: Avoid spaces, brackets, or non UTF-8 characters when naming virtual machines and their associated files.

1. Commit any snapshots.
2. Shutdown the virtual machine.
3. If you are using VirtualCenter, remove the virtual machine from the inventory but do not delete the files from disk.
4. Connect to the ESX Server host on which the virtual machine resides over SSH. For more information, see Opening a command or shell prompt (1003892).
5. Run this command to unregister the virtual machine:
   
   vmware-cmd -s unregister <path to config file> 
   
   Where <path to config file> is the path to the configuration file as determined by the vmware-cmd –lcommand. For more information on the vmware-cmd –l command, see the VMware Scripting API.
   
   For example:
   
   vmware-cmd -s unregister '/vmfs/volumes/storage1/vm1/vm1.vmx'
6. Rename the folder, .vmx file, and .vmdk (+ flat) file to match the new name.
   
   
   mv /vmfs/volumes/storage1/vm1/ /vmfs/volumes/<perhaps new storage?>/NewVM/
   cd /vmfs/volumes/<perhaps new storage?>/NewVM/
   mv vm1.vmx NewVM.vmx
   mv vm1.vmdk NewVM.vmdk
   mv vm1-flat.vmdk NewVM-flat.vmdk
7. Edit the vmx file to reflect the name of the new descriptor file.
   
   Locate the scsi0:0.fileName line, it appears similar to:
   
   scsi0:0.fileName = "vm1.vmdk"
   The edit appears similar to:
   
   scsi0:0.fileName = "NewVM.vmdk"
8. Save the file and exit.
9. Edit the .vmdk file to reflect the name of the new flat file.
   
   Locate the Extent description section of the .vmdk file. It appears similar to:
   
   # Extent description
   RW 8388608 VMFS "vm1-flat.vmdk"Edit this to:
   
   # Extent description
   RW 8388608 VMFS "NewVM-flat.vmdk"
10. Register the virtual machine:
    
    vmware-cmd -s register <path to config file>
    Where <path to config file> is the path to the configuration file determined by the ‘vmware-cmd –l’command.
    
    For example:
    vmware-cmd -s register '/vmfs/volumes/<perhaps new storage?>/NewVM/NewVM.vmx'
11. Add the virtual machine back into VirtualCenter inventory using the Virtual Infrastructure Client by browsing the data store, finding the .vmx file, right-clicking and adding it to inventory.

Note: There are some additional files that you may want to rename such as the .vmxf and the swap file name. Both of these are referenced from the .vmx file. If you decide to change either of these file names, ensure you update the .vmxfile while you are completing step 7 in the instructions.

Note: To rename a VMDK file, you can also use the command:

vmkfstools -E <oldName>.vmdk <newName>.vmdk

This command renames the content file (-flat.vmdk ) and also all references inside the metadata file (.vmdk).

Ensure to update your virtual machines .vmx file after completing this step.

Corrupt VMX File

Issues with a virtual machines VMX file can prevent it from being powered on.

For more information on VMX files, see Validating the .vmx settings of a virtual machine (1003748).

Possible errors you may see are:

• Error when deleting a snapshot: ASSERT /build/mts/release/bora-39823/bora/vmx/snapshot/snapshot_vmx.c:752
• Error when attempting to power on a virtual machine: "operation not allowed in the current state"  

If you suspect your VMX file is preventing your virtual machine from starting you can rebuild your virtual machine with a clean VMX file.

To rebuild a virtual machine:

1. Connect to VirtualCenter or your ESX Server host directly with the VI Client.
2. Remove the virtual machine from the inventory.
3. Right-click your ESX Server host and choose New Virtual Machine.
4. Select the Custom option.
5. Create the virtual machine with the same configuration it had before the migration.
6. When you get to the Select a Disk section, choose to Use an existing virtual disk, and point to the original.vmdk file(s).
7. Finish the creation of the virtual machine.
8. Power on the virtual machine.

If you VMX file has become corrupted, or the contents are missing, you can also rebuild the VMX file if you still have thevmware.log file from the last time it booted correctly.

When viewing the log file of the virtual machine, search for the '--- CONFIGURATION' section. The .vmx files is below.
You can copy this text to a text editor to create a working configuration file.

Note: Do not use an editor that inserts formatting that ESX does not understand. For example, Wordpad. If possible complete the editing on the ESX Server host.

The entries in the log file appear similar to:

Feb 28 18:11:47.799: vmx| DICT config.version = 8
Feb 28 18:11:47.799: vmx| DICT virtualHW.version = 4
Feb 28 18:11:47.799: vmx| DICT floppy0.present = TRUE

Remove the preceding section so it becomes:

config.version = 8
virtualHW.version = 4
floppy0.present = TRUE

Save it with the same name as the corrupted file and the replace the corrupted version.

Corrupt VMDK File

Issues with the VMDK file of a virtual machine can prevent it from being powered on.

Possible errors you may see are:

Failed to power on a file was not found

This may be seen when the VMFS3 virtual disk descriptor (.vmdk ) files are missing. There are no <vmname>.vmdkfiles. Only <vmname-flat>.vmdk files exist.

You can also see this error if the Extent description section of the .vmdk file is incorrect.

The log files show messages similar to:

Unable to find file

The system cannot find the file specified.

For more information on re-creating a missing descriptor file, see Recreating a missing VMDK (header) file (1002511).

VMFS Corruption

When a VMFS volume becomes inaccessible due to file system corruption, you may be able to migrate your running virtual machines off the filesystem.
Powered off virtual machines are inaccessible.

Note: The following process works if the disk blocks used for the virtual disks haven't been corrupted and the operating system is Microsoft Windows.

To migrate your virtual machine:

1. Ensure you have sufficient disk space on another VMFS volume to clone the virtual machine.
2. Connect to the virtual machine using remote desktop.
3. Download and install VMware converter inside the virtual machine.
   
   This can also be downloaded to your own laptop, desktop, or another virtual machine as long as you have network access to both the source and destination.
4. Run Converter and select Physical Machine and Local Machine.
5. Choose an ESX Server host as a destination and select a healthy VMFS.
6. Follow the wizard and wait for the process to complete.

Note: Do not you power-off or restart the virtual machine. When powered-off, the virtual machine cannot start again as the files become inaccessible.

If the operating system is not Microsoft Windows, you can use native system tools to copy the whole image to a network share.

For example, on Linux:

# dd if=/dev/sda of=/mnt/nfs/sda.dd.gz bs=1024k 

You must create a new virtual machine and boot it from CD to Linux rescue mode and restore the image:

# dd if=/mnt/nfs/sda.dd of=/dev/sda bs=1024k 

Note: This method creates only crash consistent image.

Fuente: http://www.vmware.com/pdf/vsphere4/r41/vsp_41_vm_admin_guide.pdf

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1004232

Jorge D.