One of my servers went into a hung state. We tried to reboot it but the reboot filed and the task eventually timeout. The same server was hung a few days back, storage team deleted the backup job, to kill the daily NetBackup snaps. The VM came up fine but went into the hung state again last night when the backup job created one more snap.
It appears that some orphaned snapshots were left by NetBackup API. Snapshots may be left on virtual machines if those API calls are not sent, not received, or the snapshot removal operation cannot complete successfully.
The orphaned snapshot filled out the datastore.
To resolve this problem the stale (orphaned) snapshot need to be deleted.
Firstly let’s confirm if VM is running from base disk (VM_Name.vmdk) or a snapshot disk (VM_Name-00000x.vmdk).
- Right-click the affected virtual machine.
- Click Edit Settings.
- Select the hard disk and note the datastore in which it is located.
- check if it is pointing to the base disk (VM_Name.vmdk) or a snapshot disk (VM_Name-00000x.vmdk).
Delete snapshot from the vSphere Client:
- Right-click the virtual machine and click Snapshot
- Select Take Snapshot.
- Enter a name for the snapshot and take a regular Snapshot.
- After the snapshot is taken, right-click Virtual machine and click Snapshot > Snapshot Manager
- Select the snapshot and click Delete All
NOTE: Depends on the snapshot size it may take several hours to finish.
Delete snapshot from ESXi command line:
- SSH to ESXi as root
- Run this command to list all the virtual machines running on the host:
- Run this command to list the snapshot for a particular virtual machine:
vim-cmd vmsvc/snapshot.get VMID
NOTE: VMID is obtained from the output of vim-cmd vmsvc/getallvms
- Run this command to create a snapshot:
vim-cmd vmsvc/snapshot.create VMID snapshot_Name Snapshot_Description Memory_Included Snapshot_Quiesced
Do not close the SSH session during the process
- . After snapshot is created, run this command to commit the snapshot:
vim-cmd vmsvc/snapshot.removeall VMID
Please like and share to spread the knowledge in the community.
If you want to chat with me please use Twitter: @AngrySysOps
Visit my FB page: https://www.facebook.com/AngrySysOps
Read my blog: https://angrysysops.com
Subscribe to my channel : https://www.youtube.com/channel/UCRTcKGl0neismSRpDMK_M4A