AWS Cloudwatch – watch out for outdated EC2config

I configured a Windows instance on AWS to start reporting logs and performance counters into their AWS Cloudwatch for centralised overview.

The day after I had started monitoring I found that it was consuming loads of CPU and memory and thus not normal,

Screen Shot 2016-07-14 at 12.01.01

Going into the instance, which took a while as it was so sluggish, I found that the EC2Config service was going berserk and I had to kill it.

Screen Shot 2016-07-14 at 10.53.59

Looking at the properties of the EC2 you can see that it was 2.2.12.30 and far from the version available today.

Screen Shot 2016-07-14 at 11.12.56

So if you want to enable Cloudwatch make sure that you first update the EC2config to be sure that your server will survive the burden of monitoring  ?

Firmware update of NIC makes team go degraded

I have been doing some maintenance on a Hyper-V environment and patching with Windows updates and also firmware for the hardware.

The hosts was Dell R730 with former Broadcom and now Qlogic 10 Gbit NIC´s. There are 4 10 Gbit NIC´s on every host and they are set up with one team for management and one team for VM´s.

I have done firmware update before but now there was a new version that I wanted to apply.

It tock a while before I realised that the NIC`s had changed name and thus that was why the teams was degraded, we had some discussions and the Networking guy checked his configuration more than once and we also verified the cabling on the server and then after a while I realised that the NIC actually had changed name and once adding the correct NIC´s to the teams the status changed to normal!

Screen Shot 2016-06-23 at 10.33.14

So when doing maintenece please check the status after updating firmware or the battle with the networking guys can end in misery on your side 😉

Explorations within AWS – Setting ACLs on temporary SSD store for SQL

I have been quite busy lately and not had the time to update the blog so much that I wanted but I will try to add some posts during the summer!

In the project I am right now we are setting up environments in Amazons cloud AWS. I have used their images AMI for a SQL Always On cluster that spans over two nodes.

The AMI is preinstalled with SQL and ready for incorporation in an domain and the Enterprise version can be used with the r3.2xlarge r3.4xlarge and r3.8xlarge.

Screen Shot 2016-07-15 at 23.01.47

As you can see each of them have an SSD instance storage that can be used as a temporary volume which suits SQL tempdb perfectly. Just go into the SQL configuration and point its tempdb to the temporary storage! The MSSQL service account is a domain account without administrative rights on the server so that is why I explicit set the rights for the volume…

As the AWS documentations clearly tells us:

An instance store provides temporary block-level storage for your instance. This storage is located on disks that are physically attached to the host computer. Instance store is ideal for temporary storage of information that changes frequently, such as buffers, caches, scratch data, and other temporary content

You can specify instance store volumes for an instance only when you launch it. The data in an instance store persists only during the lifetime of its associated instance. If an instance reboots (intentionally or unintentionally), data in the instance store persists. However, data in the instance store is lost under the following circumstances:

  • The underlying disk drive fails
  • The instance stops
  • The instance terminates

So I created a small powershell script that runs each time the instance boots to set ACL´s on that volume for the SQL to be able to create its tempdb files. No tempdb files = no sql service running….

And this is set up as a powershell job that triggers on the server booting

Happy AWS´ing!