After visiting a customer this friday we talked about their environment and the need to check that their cluster shared volumes in a Windows 2008 R2 SP1 Hyper-V cluster was healthy.

They have started to take host level backup but have not yet got the hardware VSS driver in place and when we checked the backup agent had got into a faulty state and locked the CSV volume in redirected access mode. When we restarted the backup agent  the status got back to online.

Aidian Finn has done a white paper about backup and CSV volumes which describes the redirection and what that can make for impact on the performance. But in short, the cluster node that is the owner will during the redirected access have sole write access to the CSV volume and that means that the other nodes in the cluster must send all storage trafic to that node.

Alan Renouf´s vCheck script is a really easy and good way to keep in a daily control  of your environment, it works for both a VMware vSphere environment and also for Hyper-V with SCVMM thanks to Jan Egil Ring. The best way is to configure it with scheduled tasks and to send a mail every morning.

What I have done is editing the Cluster Shared Volume check plugin to report the status also, as this can be quite important. It is just a minor addition to the plugin, I have also chosen to change so if you set it to 100 the CSV part will always be included in the report and not depending on a percentage space remaining.

Here is the code for the CSV Plugin:

# Start of Settings
# Free space threshold for Hyper-V Cluster Shared Volumes (value in percent)
$CSVFreeSpaceThreshold ="50"
# End of Settings

$Title = "Hyper-V Cluster Shared Volumes"
$Header ="Hyper-V Cluster Shared Volumes"
if($CSVFreeSpaceThreshold -eq 100){
     $Comments = "Hyper-V Cluster Shared Volumes information and state"
}else{
    $Comments = "Hyper-V Cluster Shared Volumes with less than $CSVFreeSpaceThreshold percent free, information and state"
}
$Display = "Table"
$Author = "Jan Egil Ring/Niklas Akerlund"
$PluginVersion = 1.1
$PluginCategory = "Hyper-V"

$FailoverClusters = $VMHostClusters | Where-Object {$_.VirtualizationPlatform -eq "HyperV"}

if ($FailoverClusters)  {

if (!(Get-Module FailoverClusters)) {
	Import-Module FailoverClusters
}

foreach ($cluster in $FailoverClusters) {
    if($CSVFreeSpaceThreshold -eq 100){
      Get-ClusterSharedVolume -Cluster $cluster.name | Select-Object -Property Name,State -ExpandProperty SharedVolumeInfo | Select-Object @{Name="Cluster";e={$cluster.name}},Name,FriendlyVolumeName,@{ Label= "State"; Expression ={if($_.RedirectedAccess){"Redirected Access"}elseif($_.MaintenanceMode){"Maintenance Mode"}else{"Online"}}},@{ Label = "Size(GB)" ; Expression = { "{0:N2}" -f ($_.Partition.Size/1024/1024/1024) } },@{ Label = "PercentFree" ; Expression = { "{0:N2}" -f ($_.Partition.PercentFree) } },@{ Label = "FreeSpace(GB)" ; Expression = { "{0:N2}" -f ($_.Partition.FreeSpace/1024/1024/1024) } },@{ Label = "UsedSpace(GB)" ; Expression = { "{0:N2}" -f ($_.Partition.UsedSpace/1024/1024/1024) } }
    }else{
        Get-ClusterSharedVolume -Cluster $cluster.name | Select-Object -Property Name,State -ExpandProperty SharedVolumeInfo | Where-Object {$_.Partition.PercentFree -lt $CSVFreeSpaceThreshold} | Select-Object @{Name="Cluster";e={$cluster.name}},Name,FriendlyVolumeName,@{ Label= "State"; Expression ={if($_.RedirectedAccess){"Redirected Access"}elseif($_.MaintenanceMode){"Maintenance Mode"}else{"Online"}}},@{ Label = "Size(GB)" ; Expression = { "{0:N2}" -f ($_.Partition.Size/1024/1024/1024) } },@{ Label = "PercentFree" ; Expression = { "{0:N2}" -f ($_.Partition.PercentFree) } },@{ Label = "FreeSpace(GB)" ; Expression = { "{0:N2}" -f ($_.Partition.FreeSpace/1024/1024/1024) } },@{ Label = "UsedSpace(GB)" ; Expression = { "{0:N2}" -f ($_.Partition.UsedSpace/1024/1024/1024) } }
    }
}
}

A customer found the bug that exists in the System Center Virtual Machine Managers function for setting a host in maintenance mode. The bug is that when you set a host in maintenance mode it will live migrate all VM´s to the next node in the cluster. This is kind of impractical when you for example want to patch the Hosts and you end up with the last host that is filled with in worst case all  VM´s on it and it takes longer time to live migrate them. The VMM bug should be fixed in the SC SP1 that is coming soon This of course depending on if you have enabled the Dynamic Optimization, that function helps you with the distribution but that will take some time before it runs.

He made a script for evacuating the host and distributing the VMs to the other nodes in the cluster. I have added a bit of logic that also set the host group to not allow dynamic optimization during the scripted evacuation.

The load balancing in my script is quite easy because right now the only thing I look at is the host memory and migrate the VM to the Host that has most memory left at the moment.

<#
.Synopsis
   A function to evacuate a host in vmm and set it in maintence mode
.DESCRIPTION
   This function migrates all your VM´s to other nodes in the cluster based on available Host memory
.EXAMPLE
   Evacuate-SCVMHost -VMHost hyp02 -HostGroup DC01
.Notes
Niklas Akerlund/Lumagate 2012-11-25
#>
function Evacuate-SCVMHost
{
    [CmdletBinding()]
    [OutputType([int])]
    Param
    (
        # What VMHost you want to set to maintmode
        [Parameter(Mandatory=$true,
                   ValueFromPipelineByPropertyName=$true,
                   Position=0)]
        $VMHost,

        # In what hostgroup the host resides 
        $HostGroup
    )
    # Get The cluster and disable the dynamic optimization during evac 
    $HostGroup = Get-SCVMHostGroup $HostGroup
    $HostGroup | Get-SCDynamicOptimizationConfiguration | Set-SCDynamicOptimizationConfiguration -ManualMode 
    $VMHost = Get-VMHost $VMHost
    # Evacuate the host
    $VMs = $VMHost | Get-VM
    foreach ($VM in $VMs){
        # Find the most apropriate Host in cluster for each VM
       $VMHostTarget = Get-SCVMHostCluster -VMHostGroup $HostGroup | Get-SCVMHost | where {$_.ComputerName -ne $VMHost.ComputerName} | Sort-Object AvailableMemory -Descending | Select-Object -First 1
       Move-SCVirtualMachine -VM $VM -VMHost $VMHostTarget  
        
    }
    # Set the host in maintmode
    Disable-VMHost -VMHost $VMHost -MoveWithinCluster
    # Enable dynamic optimization
    $HostGroup | Get-SCDynamicOptimizationConfiguration | Set-SCDynamicOptimizationConfiguration -AutomaticMode