1

I created an Azure Kubernetes cluster using Terraform. I used the recommanded azurerm_kubernetes_cluster resource, which create the nodes under the hood. As a consequence, I don't have access to the nodes as Kubernetes object. I now want to Terraform the alerting related to thoses nodes. But in order to do it, I need to have the nodes as a Terraform object (Terraform alert rule resource requires the id of the node: https://www.terraform.io/docs/providers/azurerm/r/monitor_metric_alertrule.html).

So I tried to access the nodes as Terraform data: https://www.terraform.io/docs/providers/azurerm/d/virtual_machine.html.

As stated in the previous link, we need two infos for that: the resource_group and the name of the virtual machine. The output of the azurerm_kubernetes_cluster resource gives us the resource_group, so that part is ok. But the nodes have a name generated randomly (to be more precised, one part of the name is generated randomly, the other part can be guessed from Terraform objects that we have). But as the previous link shows, there is no possibility to use a filter functionnality (such as in https://www.terraform.io/docs/providers/aws/d/ami.html) or to use a regex to match the nodes names. So the following is not possible (with the * in place of the random generated part and where part1 and part2 are known):

data "azurerm_virtual_machine" "nodes" {
  name                = "part1-*-part2"  
  resource_group_name = "${azurerm_kubernetes_cluster.this.node_resource_group}"
}

Does anyone have an id on how to unlock one of the following:

  • I can't find any explanation in the Azure AKS documentation on how the random generate part of the node name is generated (is it that random are can it be predicted?) and can't find myself by experimentation or wild guess. Does someone knows that?
  • Can we get the a list of virtual machines of a resource_group using datas in a way I didn't think of yet?
  • I can't seem to find any blog post or videos where AKS nodes alerting is done using Terraform, even with dirty tricks. Can someone provide me a link I missed to work on?

Terraform Azure provider version: 1.23.0

Terraform version: 0.10.x (required by the Azure provider 1.23.0)

1

You're coming at this the wrong way. When you create an AKS cluster, you are creating some VMs as worker nodes, however, these are not plain old VMs, you can't manage them like standalone VMs, the AKS cluster is undertaking most of the management work.

If you want to monitor the VMs, you need to do so through the AKS cluster, using AKS metrics, which include node metrics, not as standalone VMS. You can see more details of AKS metrics here - https://docs.microsoft.com/en-us/azure/azure-monitor/insights/container-insights-overview

  • I indeed set up the native monitoring with Terraform and I am able to see all the relevant metrics in the AKS metrics dashboard (CPU, memory ,etc, for nodes and containers), as you suggest. The problem I am facing initially is about alerting: there are a very few metrics about AKS cluster that can be exploited to build alerts (see docs.microsoft.com/en-us/azure/azure-monitor/platform/…). So I wanted to monitor nodes as basic virtual machines, which can allow me to access to a lot more metrics to build my alerts. – dbourcet Apr 3 at 6:51
  • Bottom line is, you can't monitor them like normal VMs, so you would need to use the metrics for the cluster, or implement your own monitoring using Prometheus etc. – Sam Cogan Apr 3 at 14:50
  • You are right. We choose for now to push metrics in our monitoring solution, which then handle the alerting, waiting for Azure to propose a lot more alerts. Do you know if using custom alerts in Azure would do the tricks, have you already defined custom alerts for Azure k8s cluster? – dbourcet Apr 4 at 6:56
  • Actually, this isn't good enough for all purposes. For instance, I want to install a network connection monitor to monitor connectivity between my cluster and resources that it depends on. I can, indeed, configure this in the portal, but I can't configure it via Terraform without the nodes' VM IDs. As near as I can discover, there is no cluster-level connection monitor, either. – Derek yesterday

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.