7

We have an application deployed to Azure IaaS VMs, served by IIS. In order to install updates, we need to take each machine out of the load balancer, one by one. Before moving to Azure, we were using Microsoft NLB which has the function to DRAIN STOP a node - by not sending new connections, but keep the existing connections open until they complete. How can we achieve the same with Azure LB?

12

The recommended way to do this is to have a custom health probe in your load balanced set. For example, you could have a simple healthcheck.html page on each of your VM's (in wwwroot for example) and direct the probe from your load balanced set to this page. As long as the probe can retrieve that page (HTTP 200), the Azure load balancer will keep sending user requests to the VM.

When you need to update a VM, then you can simply rename the healthcheck.html to a different name such as _healthcheck.html. This will cause the probe to start receiving HTTP 404 errors and will take that machine out of the load balanced rotation because it is not getting HTTP 200. Existing connections will continue to be serviced but the Azure LB will stop sending new requests to the VM.

After your updates on the VM have been completed, rename _healthcheck.html back to healthcheck.html. The Azure LB probe will start getting HTTP 200 responses and as a result start sending requests to this VM again.

Repeat this for each VM in the load balanced set.

  • Thank you @Rick, I implemented this and it works as expected! – nulldotzero Nov 3 '15 at 9:32
  • Rick - I have never setup a "custom health probe". Would you say this is the best article to follow? blogs.msdn.com/b/piyushranjan/archive/2014/01/09/… – aron Dec 3 '15 at 4:18
  • It's going to depend on your application. Each application will have different dependencies on other resources (databases, caches, web services, etc). This is a very nice post to get you started though. – Rick Rainey Dec 3 '15 at 14:26
  • 1
    @larry Silverman is this an existing tcp connection or a new one? Remember the slb doesn't close existing tcp connections so if a client was connected before it would still get connected back to the now unhealthy instance. Best way to repro this would be to run in powershell from an outside machine while($true) {iwr <url that connected to the instance>; start-sleep - seconds 10}, now go and mark the instance down, you will notice that above ps would start to fail and logging wi show request made it to the down instance, as the session persisted. Repeat same iwr with -disablekeepalive – Anirudh Goel Oct 18 '18 at 5:57
  • 1
    I stumbled upon here while searching answer for that very question, so High Five! It seems best would be for the application code to detect this shutdown event and start closing tcp connections by adding "Connection:close" header, it's an equivalent of draining tcp connections.. Also I am exploring this option serverfault.com/a/284199/83705 will keep you posted on how it looks like. – Anirudh Goel Oct 18 '18 at 18:40
1

In their documentation, Microsoft recommends using a Security Group to explicitly block the health probe. All Azure Load Balancer health probes will come from 168.63.129.16.

An example would be using an incoming NSG rule to deny 168.63.129.16 to destination of the VM NIC that you want to remove from the pool.

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.