If you are working with Azure and want to learn more there is an opportunity to go to a conference in April that is free of charge and in the center of Stockholm!
I will do a session there in the MVP theater:
Making real world Infrastructure as code in Azure, or how to make an MSP-dinosaur survive in the cloud
It’s incredibly fast change in today’s IT delivery, and for a service provider, it’s about embracing the new or risking the latest T-Rex. In this session we review how to automate and create standardized solutions in Azure where management and monitoring are included as a service. Interaction with customers through Microsoft Teams and Bots that speeds up change cases and provides quick feedback! 24/7 you can know status and costs as well as order new services that automatically end up under NOC when it reaches production status.
So now we have come to some interesting parts in my experience with our multinode Stack, In this post I will go through Marketplace management and installation of App Service RP.
To actually get something into the marketplace for tenants we need to either populate it ourselves with custom images or utilize the marketplace syndication. After deploy of the stack you need to register it to Azure.And when you have successfully done that you will get possibiltiy to download azure images that have been made available.
In the powershell tools you can find the command to upload a custom image that you might want to make available for your tenants, there is though no way to make them just available for one tenant. I utilized the superfunction Convert-WindowsImage and created a new insider Windows Server 17093 for my tenants marketplace.
then using the Azurestack tools you upload it with Add-AzsVMImage
Important: You as a Stack Admin will be responsible to make sure that the latest Images have been updated on your Marketplace, there is no automation magic that will download a new Windows Server Image once it has been released in Azure and thus keeping your marketplace up to date for tenants and they can deploy without having to patch and patch and patch before they utilize their systems…
Looking at one example my SQL VM that I have downloaded from Azure was version 14.1000 and now there is a new that I need to update to:
App Service RP
Installation of the SQL RP was very much straight forward and just follow the instructions and run (there is though one thing and that is regarding the above marketplace, you will need a Windows Server Core available for the SQL RP)
for the App Service there is a bit of more work and the prerequisite says a file server :” For production deployments, the file server must be configured to be highly available and capable of handling failures.” and a SQL server: “For production and high-availability purposes, you should use a full version of SQL Server 2014 SP2 or later, enable mixed-mode authentication, and deploy in a highly available configuration.“. Luckily You can run these in default subscription and not in a tenant subscription… But still there are some serious life cycle management that needs to be handled here with patch and update, security etc on these 6 servers (AD, FS, SQL)
After when You have those prerequisites in place it is time to start the App Service wizard, and there we had the first encounter of problems.. I had the superduper SSL cert with everything including SANs or so I thought……
Coming back to my second post you should verify a thousand and thousand times with the certificate department at your company that they do not try to take any shortcuts and miss any critical SANs. In our case an assumption that a wildcard was enough to take this rocket out of orbit for a couple of hours and getting a bit more grey hair! So make sure you have a SAN name in your certificate that says sso.appservice.<region>.<xx>.domain.yy and you will not get “The certificate dns is invalid: azurestack”
Next thing that we encountered which showed a bit later was that we deployed using the eval image (this was not to obvious in the wizard, as we had both a eval and a regular in our syndicated marketplace)
And as you can see in the Wizard during app deploy it does say latest 2016 Datacenter and nothing about eval!
Now Microsoft and the Azure team have removed the Eval from the Syndication so if you do not create your own custom image with Eval you will not get into this problems and need to mitigate this..
Once the Win 2016 Eval was removed we could get a ordinary version up on the workers by scaling their scale sets down and up, but we had to fix the controllers manually.
Also make sure that you do not lock down the SQL and fileservers vnet and public IP´s with a too narrow NSG and not let app workers and controllers reach smb shares and sql services or your app service will die and not respond!
Now we have come to my fourth post on my series of AzureStack multinode experince, previously I have been writing on the importance on the network and certificates for a successful deployment.
This post will describe the success path with updates to be applied and keeping us compliant with the support cycle, you can only be up to 3 updates behind or you will be left without support!
When we got our stack it was deployed with 1709 and during the installation process the OEM-engineers that where onsite added 1710 and 1711. When the 1712 came out we did the update ourselves. Based on our learnings it is good to have ms support standby so start with a support case because to have a more successful run of the update pack you will want to check status and space on the infrastructure VM´s and it is only in a broken glass support session connected to the privileged endpoint on a ERCS node you can get help and verify the state! Probably in future update packs they will address issues and thus making it more stable and resilient and you then will not be needing a support call but better be safe than sorry!
First as the documentation describes, upload the update files to the storage account called “updateadminaccount” where you would add a container (private) with the update:
When all the files are uploaded you can go into the update and start. When highlighting the patch the “update now” link above lights up and you can press it to start the update process!
The whole process takes about 8-12 hours depending on the size of the update and how many nodes you have in your Stack!
We had one hotfix that needed to be applied after the update of 1712 and the learning from that was that the apply failed and failed and failed but not giving a good explain on why but we learned that was because we did not RTFM and uploaded the whole folders content and not just the xml,exe and bin but also a Supplemental Notice.txt (in our defence, the update packs does not contain the text file), so removing that one and then the retry succeeded without any issues!
We have some users on-boarded on the AzureStack multi-node system and they do testing and when they forget to remove stuff and we need to make some changes we might need to remove their resources and as Stanislav mentioned there is a way to actually take over a subscription. There is though a small thing that needs to be added to the PowerShell cmds that can change a Azure Stack user subscription owner when you have a multitennant setup of your Stack, your users will have their own tenant ID´s on the subscriptions.
So to be able to access and remove resources from an user that left his subscription and resources burning you will have to do a update on both Owner and TenantId
So set up your enviroment with the AzureStack tools and PowerShell environment, connect as a cloud operator to the default subscription and then run the following::
# Get all User subs
# save the user subscription you want to take over into variable
We are doing some work in adding functionality and found an issue today! We have quite a few workloads running on a converged setup with Hyper-V nodes accessing storage from Scale-Out File Servers.
In our lab environment we have Hyper-V running on Server 2016 and these get their storage from SOFS and they currently run 2012 R2. For ordinary VM´s that has not been a problem. Now we wanted to get the VHD Scale sets (enhanced shared VHDX) set up for guest clusters running guest OS win 2016.
The documentation says not so much more than you need storage residing on a SMB or CSV volume, but when trying to add a VHDs file we get an error stating that the SOFS server does not recognize the file format of the vhd set and think it is a reserved file for windows!
Using a SMB share from a bunch of 2016 Scale Out File Servers gives no errors
So upgrade those SOFS servers first and you will not run into problem 🙂
Here is my second post on experience with our lovely AzureStack multinode that we now have running.
First of all, there is now a good doc on the AzureStack site for Datacenter Integration and it is really important to read and understand the text. It is also vital to also have the networking guys on the wagon!
For a success in the deployment you will need to have a NAT functionality within your router/fw or have a transparent proxy. The doc says it is needed for the Infra Network that is called public, it is not public reachable but do need internet access. Some routers have advanced functionality with Policy based routing that can send infra traffic to a fw and public VIP traffic directly to and from internet!
Also during deployment the BMC network will need access to internet because the deployment VM running on the HLH will need to do a AAD login and registration if the stack is not being deployed as a disconnected version.
Before you can get a deployment up and running you will need to make sure that the certificates that you ordered are rock solid! Follow the documentation and do not take any short cuts in wild card certs etc…
There is a sample cert INF template file that you can use:
I had Honolulu in an earlier release installed on a Windows Server Insider 17035 build and wanted to try the new feature that came in 17079 with inplace upgrade, that did not however work out as wanted and I had to run a clean VM instead! I have not looked more into why the upgrade failed, probably because the documentation said that it was supported from 1709…
Thanks to the insider system it is already a VHDx there that I could download and use and add it to my domain.
Once that was done I could add the honolulu msi into the VM with the magnificent Copy-VMFile cmdlet
And after a simple install I could connect to it from a Chrome browser! (IE is not supported)