Global Update Manager in Win Failover Clusters

Yesterday we experienced some issues in a large hyper-v cluster that had nodes evicting and restarting the cluster service resulting in VM´s beeing restarted on other nodes and that is not great when trying to have a High Available service for the end users.

Reading about the Global Update Manager and how it works and also getting help from the Microsoft CSS helped us getting out of the issue. In default async mode in 2012 R2 the cluster update is commited when a majority of the nodes have processed it, when reading the cluster state the nodes need to check with a majority of the nodes once again to get a valid state so this means more traffic. 

The problem is when you have a large Hyper-V  2012 R2 cluster with lots of cluster resource updates and VMM, SCOM agents hammering the cluster database function gets lots of traffic and that can be cumbersome and at last it might start going bananas and evicting hosts that does not respond in time. Your logs will start filling up with event 5377 and 1135.

GUM cluster synchronous mode

There have been some work under the covers and now the cluster database read write mode has changed to a default of synchronous mode in a Hyper-V 2016 cluster. When doing it synchron it means all nodes in the cluster will process the state and that also means that all nodes have the latest info and thus can read it locally and that means less net traffic!

Default Behaviours in Clusters

Windows Server 2012 R2

Get-Cluster | fl DatabaseReadWriteMode

DatabaseReadWriteMode : 1

Windows Server 2016

Get-Cluster | fl DatabaseReadWriteMode

DatabaseReadWriteMode : 0

ref: https://windowsprivatecloud.wordpress.com/about/configure-the-global-update-manager-gum-mode-in-wfc/

Once we changed to the DatabaseReadWriteMode 0 the cluster became stable. 

My top 10 session I was @ MS Ignite

I am right now digesting the full and exciting last week and wanted to update you on the top 10 sessions of all that I attended in person. We are all different and have different taste and here you can see what I picked out.

I had like 50 sessions in my schedule that I did not manage to attend but will try to find time to watch the recordings and I will do another follow up post with the top sessions of all from Ignite later!

GS001 – An end-to-end tour of the Microsoft developer platform

Allthough I am more of an Operations guy I did not attend the infrastructure foundation session with Corey Sanders and I am happy I went for the Scott Hanselmans instead and have an insight into how the total developer experience now is within the Microsoft space and it was a great session that i can highly recommend you check out the recording of.  

DT1001 – Voices from the top: Leaders get real on building inclusive work cultures

This year the  Ignite conference had a track on Diversity and Tech and the first day I attended this lunch-session and being a nerd-techi and attending a non-technical session was great! This session gave me some great insight on how we as an industry that is very much still male-focused and need to work on our culture values to be able to get more people in.

BRK2215 – Real World architecture considerations for Azure: how to succeed and what to avoid

This first session on the Tuesday morning gave some insights from the Fasttrack team on how to do best practices when setting up environments within Azure.

BRK3062 – Architecting Security and Governance Across your Azure Subscriptions

On this session we got a high level overview on governance work within Azure and very valuable insights in the releases that had been announced regarding policys, resource graph, cost, management groups and blueprints. Also we got an insight into the in-guestvm policy work that Michael Greene with the powershell team have enabled.

BRK2269 – WinOps: Lessons learned from Enterprise devops with Microsoft technologies

Avesome session on how to apply devops thinking within the Microsoft Technologies and IT Pro space by the DevopsGuys Stephen Thair. He had some great valuable points and case studies where devops principles had been successfully implemented.

DT1003 – Service degraded: Recognizing mental burnout in your colleagues and yourself

Another great session from the Diversity track and this time it was Sonia Cuff (Azure Ops Advocate) that presented on the health topic and how to get into control of signs of burnout within yourselves or your colleagues. She did a splendid job and I did really value being here and not in Snovers PowerShell session that this one collided with in the schedule (Some sessions can be saved for later via the recording)

BRK1094 – Accelerating your IT career

Thursday morning and the room was almost full with the one and only Ned Pyle talk about how to survive in the changing landscape and how to see and work on your career. He had several tips on how to succed. The four pillars of success that included Discipline, Technical powerhouse, Communication, Legacy.

BRK2362 – The SRE role: An unexpected journey

I have not heard Jared talk before but this was a great session on the path for him and Microsoft adopting SRE (Site Reliability Engineers) practices that Facebook, Netflix and Google have been utilizing for years. It was fun analogies and his story going from being a server hugger to a cattle farmer made most of the crowd to recognise themselves.

BRK3085 – Deep dive into Implementing governance at scale through Azure Policy

Last session before the Microsoft Ignite celebration party but the room was full and everyone wanted to get more insigt into the work with Azure Policy, Azure Resource Graph and Azure Blueprints. Always a deep dive gives that extra layer of understanding and this time was no exclusion  having the product team presenting their stuff gives that extra nudge.

DT1005 – In conversation – raising the next generation of IT pros as diversity and inclusion champions

The final day I listened to this panel that talked on an interesting topic that I can truly recommend you to watch the recording. Among the people on stage was Donovan Brown talking about the struggle to hire the right staff as a People Manager.