Britton Karon
August 11, 2017

PowerShell to Manage VMs and Warmup in a Load-balanced Environment

Britton Karon

What does that even mean? Well, some of you reading this will fall into both of these buckets:

  • You are using VMs to run a memory-heavy application such as a CMS that takes a long time to build before it begins responding to requests
  • You are using a layer-4 load balancer such as the Azure Load Balancer 

Open Systems Interconnection (OSI) Model
If you do, you may have run into a problem with deploying your app to one VM at a time. That is, your VM isn’t pulled out of the rotation like you expect it to and your users are directed to a “building” screen or they wait at a blank screen while the app loads.

Why is this happening?

Your load balancer will stop directing traffic to an unhealthy instance. However, a layer-4 load balancer is only looking for a response from the server to make that call. In the case of Azure Load Balancers that’s the HTTP 200 OK response. When your app is rebuilding:

  • Your app will not respond quickly (keeping it in the rotation waiting for the 200)
  • Your app will display a loading screen that sends a 200

Either way, your VM stays in the rotation and provides a bad experience.

Man sitting at a computer screen that is loading

My rationale

While there are possibly many ways to approach this, I chose to solve this problem using PowerShell and Windows Firewall. Why did I go this route?

  • The application I was dealing with was closed-source so I didn’t have the liberty of changing the loading behavior or overriding the response
  • The application had a long warmup time
  • There were other applications running on the server that I didn’t want to restart
  • Deployments were made via an automated pipeline with no manual steps other than approval, but the pipeline is capable of remotely executing PowerShell scripts

A Solution

The PowerShell script I ended up using contained 3 main elements

  • Windows Firewall port manipulation
  • HTTP requests
  • CPU monitoring

Close the Ports

The first step in the pipeline execute a PowerShell script that closes the ports.

Set-ExecutionPolicy -ExecutionPolicy Bypass -Scope Process -Force

You may or may not need this depending on your server’s policies. The above allowed me to run the script and all Cmdlts without having to relax the policy for all scripts.

Set-NetFirewallRule -Name IIS-WebServerRole-HTTP-In-TCP -Enabled False
Set-NetFirewallRule -Name IIS-WebServerRole-HTTPS-In-TCP -Enabled False

The above do what they look like they do, they disable HTTP and HTTPS rules in Windows Firewall. The name shown above is not the name you see in the GUI; rather it is a system name that could also be a GUID. You can track those down using

Get-NetFirewallRule -All

Rebuild the App

After the deployment has taken place, the site won’t rebuild until a request has been made. In the case of the app I was working with, the first request does the bulk of the work.

Invoke-WebRequest -Uri https://www.website.com -UseBasicParsing

This requests the page and returns the HTML. You could do this for as many pages as you’d like if you want a few key features cached ahead of time.

Let Things Settle

I chose to wait for the server to become stable before putting it back into rotation. That is, I monitor the CPU until utilization is consistently low to ensure the application is finished building.

While ($count -lt 5) {
$CPUPercent = Get-WmiObject -class win32_processor | Measure-Object -property LoadPercentage -Average | Select-Object -ExpandProperty Average
Write-Output $CPUPercent
if ($CPUPercent -lt 10) {
$count = $count + 1
Start-Sleep 30
}
}

Basically, this is a loop that gets the CPU percentage every 30 second and continues to loop until the CPU is at less than 10% utilization 5 times. You can choose the appropriate utilization threshold for your application and you could even require that they be consecutive.

Open the Ports

At this point, it’s safe to open the ports and thus add the server back into rotation.

Set-NetFirewallRule -Name IIS-WebServerRole-HTTP-In-TCP -Enabled True
Set-NetFirewallRule -Name IIS-WebServerRole-HTTPS-In-TCP -Enabled True

Happy DevOps-in!

Hopefully this comes in handy for others. I’m sure at least a handful of you out there have another approach to this. If you do, please share your ideas in the comments below!

Comment

  1. RadEditor - HTML WYSIWYG Editor. MS Word-like content editing experience thanks to a rich set of formatting tools, dropdowns, dialogs, system modules and built-in spell-check.
    RadEditor's components - toolbar, content area, modes and modules
       
    Toolbar's wrapper 
     
    Content area wrapper
    RadEditor's bottom area: Design, Html and Preview modes, Statistics module and resize handle.
    It contains RadEditor's Modes/views (HTML, Design and Preview), Statistics and Resizer
    Editor Mode buttonsStatistics moduleEditor resizer
      
    RadEditor's Modules - special tools used to provide extra information such as Tag Inspector, Real Time HTML Viewer, Tag Properties and other.
       

Subscribe to Our Blog

Get the latest blog posts sent right to your inbox so you never miss an informative post from Mercury.

 

Get Your Share On!

 

Tags

Archived Posts

Go here to find all blog post published before the new year.