HTML report checking your vSphere host configuration by powerCLI version 0.1

I have today started creating a script that is sort of a check that when installing new hosts, all of them are configured the same.

It is still very simple but yet kind of powerfull, we can easily see in the html report if some vmk nic is on the wrong IP subnet or not Jumboframes activated, the following screen dump shows the report, yes it is no fancy headlines and stuff yet 😉

I am going to work a lot more such as built in error checking and try to get some nice colors if a value differs with the other hosts, and also reporting on vSwitches and portgroups that they have the right uplinks etc, but that will have to be tomorrow or another day 🙂

# Check for Host Configuration and report
#
# Niklas Åkerlund / RTS
#

$vCenter = "vcenter.demo.local"
$AdvConf = @()
$Cluster = "Cluster1"

# We only want to get info from hosts that are online
Connect-VIServer $vCenter

$VMHosts = Get-Cluster -Name $Cluster | Get-VMHost | where {$_.ConnectionState -eq "Connected" }

# Get the cluster config 
$ClusterConf = Get-Cluster -Name $Cluster | Select-Object Name,HAEnabled,HAAdmissionControlEnabled,HAIsolationResponse,VMSwapfilePolicy,DrsEnabled,DrsMode,DrsAutomationLevel | ConvertTo-Html -Fragment


# Get Basic Conf 
$BaseConf = $VMHosts | Select-Object Name,Model,NumCPU,MemoryTotalMB,Version,Build,VMSwapfileDatastore | Sort-Object Name | ConvertTo-Html -Fragment

#adv config settings
foreach ($VMHost in $VMHosts){ 
		$into = New-Object PSObject
		Add-Member -InputObject $into -MemberType NoteProperty -Name VMHost $VMHost.Name
		$AdvScratch = Get-VMHostAdvancedConfiguration -VMHost $VMHost -Name ScratchConfig.ConfiguredScratchlocation
		$AdvScratch = [string]$AdvScratch.Values
		Add-Member -InputObject $into -MemberType NoteProperty -Name ScratchLocation $AdvScratch
		$AdvSwap = Get-VMHostAdvancedConfiguration -VMHost $VMHost -Name ScratchConfig.CurrentSwapState
		$AdvSwap = [string]$AdvSwap.Values
		Add-Member -InputObject $into -MemberType NoteProperty -Name Swapstate $AdvSwap
		$AdvSyslogRemote = Get-VMHostAdvancedConfiguration -VMHost $VMHost -Name Syslog.Remote.Hostname
		$AdvSyslogRemote = [string]$AdvSyslogRemote.Values
		Add-Member -InputObject $into -MemberType NoteProperty -Name SyslogRemote $AdvSyslogRemote
		$AdvSyslogLocal = Get-VMHostAdvancedConfiguration -VMHost $VMHost -Name Syslog.Local.DatastorePath
		$AdvSyslogLocal = [string]$AdvSyslogLocal.Values
		Add-Member -InputObject $into -MemberType NoteProperty -Name SyslogLocal $AdvSyslogLocal
		$AdvConf += $into

}

$AdvConf = $AdvConf | Sort-Object VMhost | ConvertTo-Html -Fragment

# Vmk ports and their MTU

$NetConf = $VMHosts | Get-VMHostNetworkAdapter | where {$_.Name -match "vmk"} | Select-Object VMHost,Name,IP,VMotionEnabled,FaultToleranceLoggingEnabled,ManagementTrafficEnabled,Mtu,PortGroupName | Sort-Object Name,VMHost | ConvertTo-Html -Fragment

# Create the html report from the different parts 
ConvertTo-Html -body "RTS Install documentation <p> $ClusterConf <p> $BaseConf <p> $AdvCOnf <p> $NetConf" -Title "RTS Installationscheck" | Out-File install.html

 

OS X Lion reset password and how to protect yourselves

Today i realized that i was kind of vulnerable with my Mac. Of course if anyone get the hands on your computer that is not good. Todays post will give you a little heads up and  some of you will secure your MacBooks from immediate access. There are always ways to get hold of your data but don´t do it to easy.

I had totally missed the firmware password, which is used as a security add-on that prevents any user to take your Mac and boot into rescue mode and then reset your password.

So how do you reset password on someones computer then,

Reboot the Macbook and press the “Option + R” , then you will get the recover boot

As you can see in the Utilities menu list there are some different tools, the one we want is the Terminal, and there you type “resetpassword”, without any cd or external osx media you get a root terminal to use..

You will get a fine graphical dialog asking what volume and what account you want to reset password on!

So how can we make it a bit more difficult to do this then? Still not impossible but at least more difficult and time consuming 🙂

The highlighted menu option in the first picture “Firmware Password Utility” is the one we want and set a password that is going to be asked for every time we want to do some alternative booting (Recover mode, usb hdd, DVD )

So now i have activated this and how does it work, if i boot ordinary i will not get any log in promt at boot asking me to enter firmware password, but if hold down the “option” key i will get a password prompt asking me for the firmware password, the following image shows how this looks like.

This is of course no security for your data that you store on your drive, if the evil forces want your data they can take out the hdd and connect it to another computer to get data. If you are running around with sensetive data you should also enable filevault and encrypt your profile and files. In System Preferences under Security and Privacy you can enable the FileVault. As it clearly says in the warning, if you loose your password and recovery key your data is gone! And it has to be the password that you set it up with, it will not work with a reset password.

Described in several google hits there are ways to reset the firmware password also, I have not tried those yet but i will..

I also use TrueCrypt to save files and stuff on encrypted volumes.

Recover Administrator password after some powershell on the Active Directory

Yes powershell can be used to administer your Active Directory, but you could do some serious damage also. I will show you one particular case where things can go very wrong and how to recover from it.

To use cmdlets for AD you simply start your powershell console and type

Import-Module ActiveDirectory

You get quite a few cmdlets to help you automate your user administration, if you write the following in your console it will list all of them.

Get-Command *-AD*

Well now to the problem, as a domain admin you do have some privileges and say that you want to disable some user accounts and you forget to add a searchbase or your filter does not do as you wish and in one line you have disabled all accounts in your domain, including admin. IF and i say IF you realize that misstake and quickly go in and enable the accounts again, you are safe, but if you log out of your session you wont be able to log in again with any account 🙂

Get-ADUser -Filter * | Set-ADUser -Enabled $False

try to log in as domain administrator and you will get this, on any DC (as long as your replication is working and if it is not you have other problems, trust me)

So how do we fix this then? luckily there is a way to do this and it is quite easy. You have to find a windows iso and as in this case a Win 2008 r2, start it in repair mode and start a CMD

when the command promt is started do this (i found it in another blog from Matheu the difference here is that i use net user administrator /active:yes to enable instead of change password)

  • Go to c:\windows\system32
  • Rename Utilman.exe to Utilman.exe.bak
  • Copy cmd.exe to Utilman.exe
  • Reboot on Windows
  • Do the keyboard shortcut Windows + U when on the logon screen
  • net user administrator /active:yes
  • log on with the domain admin account
  • Reboot on the DVD to put back the original Utilman.exe

Instead of panicking and try to restore your AD you can easily as i described log in again. This is of course a big security thing to consider in a virtual environment where users that have access to the virtual infrastructure but are not domain admins can manipulate virtual Domain controllers to get access to the administrator password “net user Administrator newpasswd123”. Here is a link to all net user commands.

So if i am going to do some account disabling i would include a searchbase in my Get-ADUser to not get the Administrator locked out by mistake and actually take the right OU to modify users on.

Get-ADUser -Filter * -SearchBase "OU=Employees, DC=Test, DC=local" | Set-ADUser -Enabled $False

In my test environment i used two DC´s and both the lock and unlocking replicated quite fast. There is maybe a way with the Active Directory Domain Services Recovery without having to do a restore, i will look into that and do a follow up post if i find any easy ways!

PowerCLI to the rescue, how to check all VMs for Network card type

I got a question what network card some VMs had in a datacenter, as a best practice you should use VMXNET 3 where it is possible because it gives the best performance.

So i wrote this very simple script in a few lines that do a csv export of all VMs and what kind of NIC they have, of course one could extend it with OS and stuff but that will have to be next time cause my schedule is kind of tight.

When you use the wizard to set up a Windows 2008 R2 the vSphere set an E1000 NIC by default and that is not what we want, so set up a correct template or remove this nic and add a new when installing single machines!

# Get the Virtual Network Adapter
#
# Niklas Åkerlund / RTS
$VMs = Get-VM *
$Data = @()
foreach ($VM in $VMs){
$NICs = $VM.NetworkAdapters
foreach ($NIC in $NICs) {
$into = New-Object PSObject
Add-Member -InputObject $into -MemberType NoteProperty -Name VMname $VM.Name
Add-Member -InputObject $into -MemberType NoteProperty -Name NICtype $NIC.Type
$Data += $into
}
}
$Data | Export-Csv -Path e:\temp\admna\NICs.csv -NoTypeInformation

 

Hyper-V R2 Core host networking problem in VMM 2008 R2

This Friday i helped a customer with a little problem, they have a Hyper-V cluster with 4 nodes, after a switch firmware upgrade they experienced some networking instability and in their search of failure they accidentally unchecked the Host access checkbox in the networking properties on the net, as their networking configuration did not allow a separate nic for the management we had to set it up with this enabled. Of course when they applied this change the host lost connection.

I instructed them to use the 59manager to configure the host locally and set the host access on this virtual switch, this did not help cause the server would still not respond. I went over to help them on site, the first thing we did was to remove the host from the cluster and then start to test and when we removed the virtual switch the nic started to respond to ping but as soon as we re-added it to the virtual switch it stopped working. We also tried to remove the nic config and set it to dhcp and then add it to the virtual switch with the host access enabled, which did not work either. After this when we tried to set the IP in SCONFIG we got an error that stated that there was an error and the address could not be set. I thought it might be some bug in the team networking sw or drivers so we updated those as well, but no more luck there either..

Then we found the following site that described the exact same problem, Microsoft Enterprise Networking Team , in this blog they referred to a script that clean out the whole hosts virtual networking config, “nvspcrub.js” , with the /p option. Well we had to try something so we ran the script and cleared the hosts all virtual switches.  Then we added a new virtual switch and checked the Host access and tried to set the IP in SCONFIG, Still same error with “Can not set IP address”, After this we where almost on our way to give up and reinstall the host, then we thought of one last chance to set the IP through netsh (this after reading about a bug with SCONFIG) so with the command

netsh int ip set add "Local area connection 3" static 192.168.8.139 255.255.255.0 192.168.8.254

It actually worked and the host started to respond to ping 🙂 , quite frustrating that we removed all the config and then find out it was a bug in sconfig that was the causing the error.

Now we had to re-add all virtual networking switches, this was of course a perfect job for powershell, so i wrote a script that took one of the other hosts config and created the same virtual switches on the failed host, also connecting to the NIC corresponding to the right net and vlan

# Create Virtual Networks on Host
#
# Niklas Akerlund /RTS

# Take ref nic from another host

Add-PSSnapin Microsoft.SystemCenter.VirtualMachineManager
$VMMserver = Get-VMMServer sbgvmm01

$Networks = Get-VirtualNetwork | where {$_.VMHost -eq "HYP04.desso.se" -and $_.HostBoundVlanId -ne "3750"}

$NICs = Get-VMHostNetworkAdapter | where {$_.VMHost -eq "HYP01.desso.se"}

$VMHost = Get-VMhost -ComputerName "HYP01.desso.se"

foreach($Network in $Networks){
$split = $Network.Name -split ' '
if ($split[1] -eq "1"){
$Name = $split[0] + " " + $split[1]
$match = $split[2] -match "\d+"
$vlanid = $Matches[0]
$vlanid = [int]$vlanid
}elseif ($split[1] -like "VLAN*"){
$Name = $split[0]
$match = $split[1] -match "\d+"
$vlanid = $Matches[0]
$vlanid = [int]$vlanid
}else{
$Name = $split[0] + " " + $split[1]
$match = $split[2] -match "\d+"
$vlanid = $Matches[0]
$vlanid = [int]$vlanid
}

$HostNIC = Get-VMHostNetworkAdapter -VMHost $VMHost | where {$_.ConnectionName -eq $Name}

if ($HostNIC -ne $null){

New-VirtualNetwork -Name $Network.Name -VMHost $VMHost -VMHostNetworkAdapters $HostNIC -BoundToVMHost $FALSE
Set-VMHostNetworkAdapter -VMHostNetworkAdapter $HostNIC -VLANEnabled $TRUE -VLANMode "Trunk" -VLANTrunkID $vlanid
write-host $HostNIC.ConnectionName
Write-Host $vlanid
}
}

After i ran this and the host got all it´s virtual networks back i could add the host back to the cluster again. Instead of some typo errors with manually entering all the virtual switches, with some powershell we could be sure that we got the same config as the other host already in the cluster!

One thing that i first missed was the -BoundToVMHost $FALSE in the New-VirtualNetwork which resulted that all my virtual networks had the Host Access checkbox marked and i had one NIC for each of them on my host, this of course was not what i wanted, one could think that this would be false as default but for some reason MS and the VMM team thought different, well no worries i created a small script to just update my virtual networks with that option (the script above is corrected after my mistake), so i ran:

# Update networks with BoundtoHost $false
#
# Niklas Akerlund /RTS

$Networks = Get-VirtualNetwork | where {$_.VMHost -eq "HYP01.desso.se" -and $_.HostBoundVlanId -ne "3750"}

foreach ($Network in $Networks){
Set-VirtualNetwork -VirtualNetwork $Network -BoundToVMHost $false
}

Where the network with the VLAN 3750 was the one i wanted the host access to stay because it was the management nic of the host.

Hot-add CPU and Memory in a Win 2008 Datacenter with SQL in vSphere

I have today tested how it works to hot-add both memory and vCPU to a virtual machine running Windows 2008 R2 Datacenter Edition, this machine also has SQL 2008 R2 Enterprise edition installed.

First i had to enable hot-plug in the virtual machine, there is of course two ways to do this, either via the GUI or

the powerCLI way:

$vmConfigSpec = New-Object VMware.Vim.VirtualMachineConfigSpec
$mem = New-Object VMware.Vim.optionvalue
$mem.Key="mem.hotadd"
$mem.Value="true"
$vmConfigSpec.extraconfig += $mem
$cpu = New-Object VMware.Vim.optionvalue
$cpu.Key="vcpu.hotadd"
$cpu.Value="true"
$vmConfigSpec.extraconfig += $cpu

$vm = Get-VM TempNHot | Get-View
$vm.ReconfigVM_Task($vmConfigSpec)

when this has been executed on the particular VM i can then when it is running use powerCLI again and add resources to it.

Get-VM TempNHot | Set-VM -MemoryMB 3072 -NumCpu 4 -Confirm:$false

According to Microsoft documentation it is only supported to do hot-add of memory and CPU in Datacenter edition of Windows, Also regarding SQL you have to use either Enterprise or Datacenter edition to also get it working into the application.

To check that the new resources where used i tested with the SQLstress application to get some load on the SQL server and check the taskmgr, but it did not show that the load where spreading to the new added vCPU, after some research i found out that the SQL server do not start to use new hardware right away, it need to be reconfigured to schedule load on additional CPU´s, so there is a little manual intervention but no downtime on the server!

before:

So i started a query in SQL Management Studio and wrote

RECONFIGURE;
GO

After :

In the taskmanager i could now see that all four vCPU where equally loaded by the SQL server.

Maybe it is not so common to have to do this but if you set up a large Tier-1 SQL server in a virtual world you surely want to be able to hot-add resources when it is loaded. Think of the advantages that this brings when you actually both can add memory and cpu resources without any downtime!

In a virtualization world we always recommend our customers to buy Windows Datacenter licenses on their hosts so using it on a VM will not add any extra cost. The SQL server is of course quite a price jump from standard to enterprise but if your big SQL server uses more than 64 GB ram you will still need to use Enterprise licenses 🙂

 

TEC 2011 Successfully implement and Transition into Hyper-V Session

Some summary based on my own session i held at The Experts Conference 2011 in Frankfurt yesterday. I think it was about 40 people in the crowd. The TEC is about 350 attendees total.

When i have checked with Quest if i can put my whole presentation here i will do an updated posting, i will put some points what i think is crucial when setting up a new vitualization platform.

  • Assessment and Consolidation Planning
  • Design and Testing
  • Migration and Optimization
  • Capacity Planning and Performance follow up

When deciding for a new virtualization platform, no matter if it is the first or you are going to exchange an existing, there are some steps that need to be considered. First you have to know what you are running in your datacenter, what kind of operating systems and what kind of applications, also you must get a workload profile for those servers to know what their demands are. If you do not do your homework and plan for the load you will surely get some beating from your organization when you have virtualized the servers and they run like crap. As tools you can use the Microsoft Assessment and Planning Toolkit or if you already use SCCM/SCOM you will both have inventory and performance data. Another thing to consider when planning is licensing, in a big consolidation you can save quite a much money when using Datacenter licensing on the hosts.

Design your platform to be modular and easily expanded. Do serious deciding on what your boundaries are both technically and financially this should be done in a workshop with application owners and management and then documented. Do not forget about managing and monitoring software. Another thing to rethink is how to take backup in your platform, with Hyper-V integration tools you can take consistent backups with VSS snapshot support into the VMs, we recommend our customers to take backup on host level for quicker RTO. When you have decided for a platform you can do a PoC to test your decisions and see that it works as expected, Many hardware manufacturers do lend out hw for you to test for a limited time to evaluate.

When the platform is set up and correctly configured you want to do some hardware and load testing, there are several tools for this. Memtest, IOmeter, SQLiosim, Exchange jetstress, Exchange Load Gen and others. The most important thing to consider here is that you want to check that your new platform can handle the load you did measure and predict for in the analysis and design. Also test fail-over functionality so that all hardware and software works as expected when a PSU or a network cable brakes. After all testing has successfully been made you want to document this for later so you have a validation document signed.

After you have a platform set up and tested you want to start migrate and optimize workloads into this. There are some tools that can be used for this, SCVMM, disk2vhd and Quest vConverter for example. One thing to consider when doing migration is to look back at the analysis of the workloads to set the right amount of resources, both virtual processors/vm ram and vhd disk files with the partitions inside (for best performance we would like to use fixed or pass-through disks). When optimizing after migration you want to clean out hidden devices and services/software that was used for the machine in the physical world but do not have a purpose anymore!

When all your machines are migrated we want to continuously check the performance and capacity so you can prepare and implement additional host resources before it runs out. You can use SCOM/SCVMM if you have it in your environment, another great performance tool is the PAL (Performance Analysis of Logs ) that you can use in conjuction with performance counters and logman to schedule datasets on your Hyper-V Core host servers, also there is a product from Quest, vFoglight.

the last slide i had a strip from Dilbert that i find quite funny, statement though: WE DO NOT LEAVE OUR CUSTOMERS AS DOGBERT DOES AFTER A virtualization Project 😛

links

MAP 6.0

Memtest

PAL

Performance tuning win 2008 R2 SP1

 

Configure VM settings and vmdk´s with powerCLI

I want to share my latest automation scripting, i am in a project where we are in-sourcing from a hosting company. We have connected the hosts to the outsourcers NFS share, of course with powerCLI, when doing it this way i get the datastores on all servers in our cluster, without the risk of differences between the hosts datastores.

# Create NFS shares on all hosts
#
# Niklas Åkerlund /RTS
$NFSdatas = Import-Csv -Path "nfsdatastores.csv" -Delimiter ";"
$VIHosts = get-cluster -Name Cluster1 | get-vmhost | where {$_.ConnectionState -eq "Connected"}
foreach ($VIHost in $VIHosts){
foreach ($NFSdata in $NFSdatas){
$NFSHost = $NFSdata.Host
$NFSshare = $NFSdata.Share
$NFSShareName = $NFSdata.ShareName
if (($VIHost | Get-Datastore | where {$_.Name -eq $NFSShareName -and $_.type -eq "NFS"}-ErrorAction SilentlyContinue) -eq $null){
Write-Host "Monterar NFSstore $($NFSShareName) på $($VIHost)"
New-Datastore -Nfs -VMHost $VIHost -Name $NFSShareName -Path $NFSshare -NfsHost $NFSHost
}
}
}

Now when we have this in place, during the transitions the hosting company shut down the VMs on their hosts that we are going to take over.  And we add the VM to the inventory on our vCenter, when doing this the vmdk got a different datastore id in the config, also some settings should be updated to the corporate standard for the virtualization platform at the customer.

# Script to update VM with vmdk and right settings
#
# Argument in is VM name
# Niklas Akerlund / RTS AB 2011

$VMname = $args[0]
if ($VMname -ne $null){
$VM = Get-VM $VMname
$Datastore = Get-Datastore -VM $VM
$HDDs = Get-Harddisk -VM $VM

# Remove incorrect hdd referenes

Remove-HardDisk -HardDisk $HDDs -Confirm:$false

foreach ($HDD in $HDDs){
$HDDname = $HDD.Filename
$HDDsNames = $HDDname.Split("/")
$count = $HDDsNames.count
$VMdkName = $HDDsNames[$count-1]
#Write-Host $VMdkName
$diskpath = "[" + $Datastore.Name + "] " + $VM.Name + "/" + $VMdkName

#Write-Host $diskpath

New-HardDisk -VM $VM -DiskPath $diskpath
}

# Reconfigure VM Settings

$spec = new-object VMware.Vim.VirtualMachineConfigSpec
$spec.MemoryAllocation = New-Object VMware.Vim.ResourceAllocationInfo
$spec.MemoryAllocation.Limit = -1
$spec.CpuAllocation = New-Object VMware.Vim.ResourceAllocationInfo
$spec.CpuAllocation.Limit = -1
$spec.tools = New-Object VMware.Vim.ToolsConfigInfo
$spec.tools.toolsUpgradePolicy = "manual"
$spec.swapPlacement = "inherit"

$VM = $VM | get-view
$VM.ReconfigVM_Task($spec)
}

After this we start up the VM and later we do a storage vMotion of the VM to the customers FC-SAN

Get-VM theVM | Move-VM -Datastore fcdatastore1

Host Profiles and vmkernel ports with Jumbo Frames MTU 9000

Today i have found a limitation using host profiles and this together with a vmkernel port that has Mtu 9000 activated. maybe it has not been a requirement when designing the host profiles?

We set up the reference host with 4 vmkernel ports, one for management, one for vmotion, one for FT and one for NFS. The port that we wanted to use Jumbo frames for was the vmotion port.

As i wrote in an earlier post, I used the powerCLI to configure the Mtu for the actual vmkernel port Get-VmhostNetworkAdapter -Name vmk1 | Set-VmhostNetworkAdapter -Mtu 9000.

Then i add this host as an reference host in the Host Profiles and attach it to the cluster. Adding a new host and then Apply profile, creates all our vmkernel ports correctly but when checking what Mtu the vmotion vmkernel port got, it is created with the default Mtu of 1500. This is not so good because i do not want to use several different ways to configure and i want to be able to trust the Host Profiles solution. The only vmkernel port that was created before applying host profiles was the management port so it has nothing to do with editing exisiting. So the result is that i need to after applying a host profile, run a powerCLI command to edit the Mtu.

Strangely no matter if the Mtu is 9000 or 1500 the hosts are compliant in the GUI..

This applies to vSphere 4.1 u1 (i do not know how this behaves in vSphere 5)

Conclusion of this is that I have to think a bit more about using the Host profiles. If it is not fully implemented then it is not usable to get uniform hosts.

Edit vmkernel port MTU on distributed switches – using PowerCLI

According to the KB 1038827 “Enabling Jumbo Frames for VMkernel ports in a virtual distributed switch”, VMware says that you have to recreate the vmkernel port to set the MTU for jumbo frames. This is not true if you use powerCLI, I do not know exactly how it is done beneath the hood but it is very easy to configure using quite a few lines scripting..  By the way, there is no way in the GUI to edit this.

$cred = Get-Credential
Connect-VIServer ESXhost.test.loc -credential $cred

Get-VMHostNetworkadapter -name vmk2 | Set-VMHostnetworkadapter -Mtu 9000

Get-VMHostnetworkadapter -name vmk2 | ft Mtu

Setting the Mtu on the vmkernel port is basically not different using a standard vSwitch or a distributed vSwitch.

Of course you can connect to a vcenter and add a foreach loop to set the Mtu for more than one host vmkernel port.