Rescue Missing VM with unmerged snapshots on HyperV/VMM
Today I have been at a customer and helping them in a SC VMM issue where the VM went missing during a storage migration from one CSV volume to another. We are working on a project to upgrade this environment from Hyper-V 2008 R2 Sp1 to 2012 and VMM 2012 SP1 but this issue is from the VMM 2012.
On the vm there were three avhd files from snaphots taken at different times and the files was quite big.. the virtual machine original vhd was 53 GB and the snapshot differencing disks where about 40 GB each. So quite a bit of data to merge. The customer removed the snapshots but was a bit too eager to start the move so the vm had not merged the snapshots files before they started the migration.
During the migration the VMM server lost connection for a very short moment to the host and the result was that the vm got missing in the VMM console and also in the Hyper-V manager. The job had removed the VM´s configuration file but not moved it to the new location so what was left in the virtual machine folder was just the vhd and avhd files.
To find which avhd file that was the latest and find the snapshot-chain we used the “inspect disk” and “inspect parent” in Hyper-V manager, in Win 2008 R2 Sp1 it does not show the avhd file by default so you have to copy and paste the file name and then it works. As the files was in a merge process it was not obvious which file was the latest, in a normal case it is quite obvious to see in the date properties of the avhd files.
When we found which was the right avhd file we created a new VM and attached the avhd file, same procedure as with the inspect disk, copy and paste the file name in the field. If you would take the wrong avhd file and start the VM the differencing chain would break and all changes in the files after would be lost!
Now We could start the VM and verify that we got all data until the latest snapshot and that the customers user not had lost any data! As it now was a new VM with a new network card we had to enter the IP settings on that network connection.
The easiest way we thought of to get the VM to merge the avhd´s was to use the “Clone” feature in the VMM console.
And when the clone of the VM was finished that job had also merged the disks and the virtual machine started without any errors and avhd files 🙂
I have made two blog posts a while ago about checking if a vm have snapshots before migrating it to another cluster and also a post about how you can extend the console with info on what virtual machines have snapshots to easily have control!
I have said it before and will emphasize it again, do not use snapshots as backup solution and remove them as soon as you are done as they do impact on performance !
I look forward to when we have the environment on hyper-v 2012 where the live-merge feature exist and also later the R2 feature with the live clone!