Zenn
😔

【MS Learn】Azure Kubernetes Service (AKS) でコンピューティング コストを最適化する を一部やってみた

2023/10/05に公開
Azure
AKS
tech

はじめに

やってみたシリーズです。

今日はこちら
https://learn.microsoft.com/ja-jp/training/modules/aks-optimize-compute-costs/

ただし、プレビュー機能のところは諸事情によりやってないので、ゼロスケーリングを有効にするところのみやりました。

手を動かすところだけ記述していきます。
番号とかは飛び飛びになりますので、ご本家様と照らし合わせてみていただけると幸いです。

Azure Cloud Shellのbashで実行しています。

ユニット3: 演習 - AKS クラスターで複数のノードを構成し、ゼロへのスケーリングを有効にする

新しいリソース グループを作成する

2-4. 変数を登録して、rg-akscostsaving という新しいリソース グループを作成します。
その際にクラスター名をコピーしておくのを忘れずに。

Azure Cloud Shell
$ REGION_NAME=eastus
RESOURCE_GROUP=rg-akscostsaving
AKS_CLUSTER_NAME=akscostsaving-$RANDOM
$ echo $AKS_CLUSTER_NAME
akscostsaving-5733
$ az group create \
    --name $RESOURCE_GROUP \
    --location $REGION_NAME
{
  "id": "/subscriptions/xxxxxxxx-xxxxxxxxx-xxxx-xxxxxxxxxxxx/resourceGroups/rg-akscostsaving",
  "location": "eastus",
  "managedBy": null,
  "name": "rg-akscostsaving",
  "properties": {
    "provisioningState": "Succeeded"
  },
  "tags": null,
  "type": "Microsoft.Resources/resourceGroups"
}

AKS クラスターを作成する

  1. バージョン番号を取得および格納します。
Azure Cloud Shell
$ VERSION=$(az aks get-versions \
    --location $REGION_NAME \
    --query 'orchestrators[?!isPreview] | [-1].orchestratorVersion' \
    --output tsv)

と、取得できたと思ってたんですが、version取得のJSONのフォーマットが変わっているのか、このコマンドでは取得できてませんでした。
なので、2023/10/05時点でのリージョンで使用できる、プレビューではない最新バージョンを自力で取得します。

Azure Cloud Shell
$ az aks get-versions --location $REGION_NAME
{
  "values": [
    {
      "capabilities": {
        "supportPlan": [
          "KubernetesOfficial"
        ]
      },
      "isPreview": true,
      "patchVersions": {
        "1.28.0": {
          "upgrades": []
        }
      },
      "version": "1.28"
    },
    {
      "capabilities": {
        "supportPlan": [
          "KubernetesOfficial"
        ]
      },
      "isPreview": null,
      "patchVersions": {
        "1.25.11": {
          "upgrades": [
            "1.26.6",
            "1.26.3"
          ]
        },
        "1.25.6": {
          "upgrades": [
            "1.26.6",
            "1.26.3",
            "1.25.11"
          ]
        }
      },
      "version": "1.25"
    },
    {
      "capabilities": {
        "supportPlan": [
          "KubernetesOfficial",
          "AKSLongTermSupport"
        ]
      },
      "isPreview": null,
      "patchVersions": {
        "1.27.1": {
          "upgrades": [
            "1.27.3"
          ]
        },
        "1.27.3": {
          "upgrades": []
        }
      },
      "version": "1.27"
    },
    {
      "capabilities": {
        "supportPlan": [
          "KubernetesOfficial"
        ]
      },
      "isPreview": null,
      "patchVersions": {
        "1.26.3": {
          "upgrades": [
            "1.27.3",
            "1.27.1",
            "1.26.6"
          ]
        },
        "1.26.6": {
          "upgrades": [
            "1.27.3",
            "1.27.1"
          ]
        }
      },
      "version": "1.26"
    }
  ]
}

1.27っぽいので、その値をVERSION変数に入れておきます。

Azure Cloud Shell
$ VERSION=1.27
  1. システム ノード プールに 2 つノードがある AKS クラスターを作成します。
Azure Cloud Shell
$ az aks create \
    --resource-group $RESOURCE_GROUP \
    --name $AKS_CLUSTER_NAME \
    --location $REGION_NAME \
    --kubernetes-version $VERSION \
    --node-count 2 \
    --load-balancer-sku standard \
    --vm-set-type VirtualMachineScaleSets \
    --generate-ssh-keys
{
  "aadProfile": null,
  "addonProfiles": null,
  "agentPoolProfiles": [
    {
      "availabilityZones": null,
      "count": 2,
      "creationData": null,
      "currentOrchestratorVersion": "1.27.3",
      "enableAutoScaling": false,
      "enableEncryptionAtHost": false,
      "enableFips": false,
      "enableNodePublicIp": false,
      "enableUltraSsd": false,
      "gpuInstanceProfile": null,
      "hostGroupId": null,
      "kubeletConfig": null,
      "kubeletDiskType": "OS",
      "linuxOsConfig": null,
      "maxCount": null,
      "maxPods": 110,
      "minCount": null,
      "mode": "System",
      "name": "nodepool1",
      "nodeImageVersion": "AKSUbuntu-2204gen2containerd-202309.06.0",
      "nodeLabels": null,
      "nodePublicIpPrefixId": null,
      "nodeTaints": null,
      "orchestratorVersion": "1.27",
      "osDiskSizeGb": 128,
      "osDiskType": "Managed",
      "osSku": "Ubuntu",
      "osType": "Linux",
      "podSubnetId": null,
      "powerState": {
        "code": "Running"
      },
      "provisioningState": "Succeeded",
      "proximityPlacementGroupId": null,
      "scaleDownMode": null,
      "scaleSetEvictionPolicy": null,
      "scaleSetPriority": null,
      "spotMaxPrice": null,
      "tags": null,
      "type": "VirtualMachineScaleSets",
      "upgradeSettings": {
        "drainTimeoutInMinutes": null,
        "maxSurge": null
      },
      "vmSize": "Standard_DS2_v2",
      "vnetSubnetId": null,
      "workloadRuntime": null
    }
  ],
  "apiServerAccessProfile": null,
  "autoScalerProfile": null,
  "autoUpgradeProfile": {
    "nodeOsUpgradeChannel": "NodeImage",
    "upgradeChannel": null
  },
  "azureMonitorProfile": null,
  "azurePortalFqdn": "akscostsav-rg-akscostsaving-a058ec-pih3hdk9.portal.hcp.eastus.azmk8s.io",
  "currentKubernetesVersion": "1.27.3",
  "disableLocalAccounts": false,
  "diskEncryptionSetId": null,
  "dnsPrefix": "akscostsav-rg-akscostsaving-a058ec",
  "enablePodSecurityPolicy": null,
  "enableRbac": true,
  "extendedLocation": null,
  "fqdn": "akscostsav-rg-akscostsaving-a058ec-pih3hdk9.hcp.eastus.azmk8s.io",
  "fqdnSubdomain": null,
  "httpProxyConfig": null,
  "id": "/subscriptions/xxxxxxxx-xxxxxxxxx-xxxx-xxxxxxxxxxxx/resourcegroups/rg-akscostsaving/providers/Microsoft.ContainerService/managedClusters/akscostsaving-5733",
  "identity": {
    "delegatedResources": null,
    "principalId": "cdf776e5-03d3-4081-931d-86625e7917ad",
    "tenantId": "cb9bb346-c037-4fb2-a3ff-dd23544753ea",
    "type": "SystemAssigned",
    "userAssignedIdentities": null
  },
  "identityProfile": {
    "kubeletidentity": {
      "clientId": "96ac9e59-12bb-4ad0-b963-a171d23621ab",
      "objectId": "c42fea9c-892a-461d-a454-20289cc73faa",
      "resourceId": "/subscriptions/xxxxxxxx-xxxxxxxxx-xxxx-xxxxxxxxxxxx/resourcegroups/MC_rg-akscostsaving_akscostsaving-5733_eastus/providers/Microsoft.ManagedIdentity/userAssignedIdentities/akscostsaving-5733-agentpool"
    }
  },
  "kubernetesVersion": "1.27",
  "linuxProfile": {
    "adminUsername": "azureuser",
    "ssh": {
      "publicKeys": [
        {
          "keyData": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCtlQI5JNtID9FT9ei748fLS8yiwL94XmBPoXsxv9UGFe3ZilBzAtPUara5AXKVHDiHcPyLuXTq6iLbbC1tvVidf1BsH/lwFFSjz/z7aSAS2I84dAu5E/+AYg6azipYYtWzcCwIPfhaP0FqgtEOb0+Cvf1Dg*******lg2qY0PdfrTU7Pf5cpCK46+zLhqQL7HpZNi/CHQrtU5TiSU6gU2gUy0/tOvlvUh4T+ipR+yH51Y2XF5v0Szy8c1brSOJ4VpO8sdxLycSw2QYOMbuv0Pxjg8YIv5yK*******p8QwOgYj1BeEdpUr4Lz2/ZuFlJoTNcPQSkxCjVI0VtRF"
        }
      ]
    }
  },
  "location": "eastus",
  "maxAgentPools": 100,
  "name": "akscostsaving-5733",
  "networkProfile": {
    "dnsServiceIp": "10.0.0.10",
    "ipFamilies": [
      "IPv4"
    ],
    "loadBalancerProfile": {
      "allocatedOutboundPorts": null,
      "effectiveOutboundIPs": [
        {
          "id": "/subscriptions/xxxxxxxx-xxxxxxxxx-xxxx-xxxxxxxxxxxx/resourceGroups/MC_rg-akscostsaving_akscostsaving-5733_eastus/providers/Microsoft.Network/publicIPAddresses/9265fb3b-****-48ed-9df1-cd62390879d3",
          "resourceGroup": "MC_rg-akscostsaving_akscostsaving-5733_eastus"
        }
      ],
      "enableMultipleStandardLoadBalancers": null,
      "idleTimeoutInMinutes": null,
      "managedOutboundIPs": {
        "count": 1,
        "countIpv6": null
      },
      "outboundIPs": null,
      "outboundIpPrefixes": null
    },
    "loadBalancerSku": "Standard",
    "natGatewayProfile": null,
    "networkDataplane": null,
    "networkMode": null,
    "networkPlugin": "kubenet",
    "networkPluginMode": null,
    "networkPolicy": null,
    "outboundType": "loadBalancer",
    "podCidr": "10.244.0.0/16",
    "podCidrs": [
      "10.244.0.0/16"
    ],
    "serviceCidr": "10.0.0.0/16",
    "serviceCidrs": [
      "10.0.0.0/16"
    ]
  },
  "nodeResourceGroup": "MC_rg-akscostsaving_akscostsaving-5733_eastus",
  "oidcIssuerProfile": {
    "enabled": false,
    "issuerUrl": null
  },
  "podIdentityProfile": null,
  "powerState": {
    "code": "Running"
  },
  "privateFqdn": null,
  "privateLinkResources": null,
  "provisioningState": "Succeeded",
  "publicNetworkAccess": null,
  "resourceGroup": "rg-akscostsaving",
  "securityProfile": {
    "azureKeyVaultKms": null,
    "defender": null,
    "imageCleaner": null,
    "workloadIdentity": null
  },
  "servicePrincipalProfile": {
    "clientId": "msi",
    "secret": null
  },
  "sku": {
    "name": "Base",
    "tier": "Free"
  },
  "storageProfile": {
    "blobCsiDriver": null,
    "diskCsiDriver": {
      "enabled": true
    },
    "fileCsiDriver": {
      "enabled": true
    },
    "snapshotController": {
      "enabled": true
    }
  },
  "supportPlan": "KubernetesOfficial",
  "systemData": null,
  "tags": null,
  "type": "Microsoft.ContainerService/ManagedClusters",
  "upgradeSettings": null,
  "windowsProfile": null,
  "workloadAutoScalerProfile": {
    "keda": null,
    "verticalPodAutoscaler": null
  }
}
  1. 新しいクラスターのノード プールを一覧表示します。
Azure Cloud Shell
$ az aks nodepool list --resource-group $RESOURCE_GROUP --cluster-name $AKS_CLUSTER_NAME
[
  {
    "availabilityZones": null,
    "count": 2,
    "creationData": null,
    "currentOrchestratorVersion": "1.27.3",
    "enableAutoScaling": false,
    "enableEncryptionAtHost": false,
    "enableFips": false,
    "enableNodePublicIp": false,
    "enableUltraSsd": false,
    "gpuInstanceProfile": null,
    "hostGroupId": null,
    "id": "/subscriptions/xxxxxxxx-xxxxxxxxx-xxxx-xxxxxxxxxxxx/resourcegroups/rg-akscostsaving/providers/Microsoft.ContainerService/managedClusters/akscostsaving-5733/agentPools/nodepool1",
    "kubeletConfig": null,
    "kubeletDiskType": "OS",
    "linuxOsConfig": null,
    "maxCount": null,
    "maxPods": 110,
    "minCount": null,
    "mode": "System",
    "name": "nodepool1",
    "nodeImageVersion": "AKSUbuntu-2204gen2containerd-202309.06.0",
    "nodeLabels": null,
    "nodePublicIpPrefixId": null,
    "nodeTaints": null,
    "orchestratorVersion": "1.27",
    "osDiskSizeGb": 128,
    "osDiskType": "Managed",
    "osSku": "Ubuntu",
    "osType": "Linux",
    "podSubnetId": null,
    "powerState": {
      "code": "Running"
    },
    "provisioningState": "Succeeded",
    "proximityPlacementGroupId": null,
    "resourceGroup": "rg-akscostsaving",
    "scaleDownMode": null,
    "scaleSetEvictionPolicy": null,
    "scaleSetPriority": null,
    "spotMaxPrice": null,
    "tags": null,
    "type": "Microsoft.ContainerService/managedClusters/agentPools",
    "typePropertiesType": "VirtualMachineScaleSets",
    "upgradeSettings": {
      "drainTimeoutInMinutes": null,
      "maxSurge": null
    },
    "vmSize": "Standard_DS2_v2",
    "vnetSubnetId": null,
    "workloadRuntime": null
  }
]

ノード プールを追加する

  1. 2 つ目のノード プールを追加します。
Azure Cloud Shell
$ az aks nodepool add \
    --resource-group $RESOURCE_GROUP \
    --cluster-name $AKS_CLUSTER_NAME \
    --name batchprocpl \
    --node-count 2
{
  "availabilityZones": null,
  "count": 2,
  "creationData": null,
  "currentOrchestratorVersion": "1.27.3",
  "enableAutoScaling": false,
  "enableEncryptionAtHost": false,
  "enableFips": false,
  "enableNodePublicIp": false,
  "enableUltraSsd": false,
  "gpuInstanceProfile": null,
  "hostGroupId": null,
  "id": "/subscriptions/xxxxxxxx-xxxxxxxxx-xxxx-xxxxxxxxxxxx/resourcegroups/rg-akscostsaving/providers/Microsoft.ContainerService/managedClusters/akscostsaving-5733/agentPools/batchprocpl",
  "kubeletConfig": null,
  "kubeletDiskType": "OS",
  "linuxOsConfig": null,
  "maxCount": null,
  "maxPods": 110,
  "minCount": null,
  "mode": "User",
  "name": "batchprocpl",
  "nodeImageVersion": "AKSUbuntu-2204gen2containerd-202309.06.0",
  "nodeLabels": null,
  "nodePublicIpPrefixId": null,
  "nodeTaints": null,
  "orchestratorVersion": "1.27.3",
  "osDiskSizeGb": 128,
  "osDiskType": "Managed",
  "osSku": "Ubuntu",
  "osType": "Linux",
  "podSubnetId": null,
  "powerState": {
    "code": "Running"
  },
  "provisioningState": "Succeeded",
  "proximityPlacementGroupId": null,
  "resourceGroup": "rg-akscostsaving",
  "scaleDownMode": "Delete",
  "scaleSetEvictionPolicy": null,
  "scaleSetPriority": null,
  "spotMaxPrice": null,
  "tags": null,
  "type": "Microsoft.ContainerService/managedClusters/agentPools",
  "typePropertiesType": "VirtualMachineScaleSets",
  "upgradeSettings": {
    "drainTimeoutInMinutes": null,
    "maxSurge": null
  },
  "vmSize": "Standard_DS2_v2",
  "vnetSubnetId": null,
  "workloadRuntime": null
}
  1. 新しいクラスター内の新しいノード プールを一覧表示します。
Azure Cloud Shell
$ az aks nodepool list --resource-group $RESOURCE_GROUP --cluster-name $AKS_CLUSTER_NAME
[
  // あとで作ったノード
  {
    "availabilityZones": null,
    "count": 2,
    "creationData": null,
    "currentOrchestratorVersion": "1.27.3",
    "enableAutoScaling": false,
    "enableEncryptionAtHost": false,
    "enableFips": false,
    "enableNodePublicIp": false,
    "enableUltraSsd": false,
    "gpuInstanceProfile": null,
    "hostGroupId": null,
    "id": "/subscriptions/xxxxxxxx-xxxxxxxxx-xxxx-xxxxxxxxxxxx/resourcegroups/rg-akscostsaving/providers/Microsoft.ContainerService/managedClusters/akscostsaving-5733/agentPools/batchprocpl",
    "kubeletConfig": null,
    "kubeletDiskType": "OS",
    "linuxOsConfig": null,
    "maxCount": null,
    "maxPods": 110,
    "minCount": null,
    "mode": "User",
    "name": "batchprocpl",
    "nodeImageVersion": "AKSUbuntu-2204gen2containerd-202309.06.0",
    "nodeLabels": null,
    "nodePublicIpPrefixId": null,
    "nodeTaints": null,
    "orchestratorVersion": "1.27.3",
    "osDiskSizeGb": 128,
    "osDiskType": "Managed",
    "osSku": "Ubuntu",
    "osType": "Linux",
    "podSubnetId": null,
    "powerState": {
      "code": "Running"
    },
    "provisioningState": "Succeeded",
    "proximityPlacementGroupId": null,
    "resourceGroup": "rg-akscostsaving",
    "scaleDownMode": "Delete",
    "scaleSetEvictionPolicy": null,
    "scaleSetPriority": null,
    "spotMaxPrice": null,
    "tags": null,
    "type": "Microsoft.ContainerService/managedClusters/agentPools",
    "typePropertiesType": "VirtualMachineScaleSets",
    "upgradeSettings": {
      "drainTimeoutInMinutes": null,
      "maxSurge": null
    },
    "vmSize": "Standard_DS2_v2",
    "vnetSubnetId": null,
    "workloadRuntime": null
  },
  // 先に作ったノード
  {
    "availabilityZones": null,
    "count": 2,
    "creationData": null,
    "currentOrchestratorVersion": "1.27.3",
    "enableAutoScaling": false,
    "enableEncryptionAtHost": false,
    "enableFips": false,
    "enableNodePublicIp": false,
    "enableUltraSsd": false,
    "gpuInstanceProfile": null,
    "hostGroupId": null,
    "id": "/subscriptions/xxxxxxxx-xxxxxxxxx-xxxx-xxxxxxxxxxxx/resourcegroups/rg-akscostsaving/providers/Microsoft.ContainerService/managedClusters/akscostsaving-5733/agentPools/nodepool1",
    "kubeletConfig": null,
    "kubeletDiskType": "OS",
    "linuxOsConfig": null,
    "maxCount": null,
    "maxPods": 110,
    "minCount": null,
    "mode": "System",
    "name": "nodepool1",
    "nodeImageVersion": "AKSUbuntu-2204gen2containerd-202309.06.0",
    "nodeLabels": null,
    "nodePublicIpPrefixId": null,
    "nodeTaints": null,
    "orchestratorVersion": "1.27",
    "osDiskSizeGb": 128,
    "osDiskType": "Managed",
    "osSku": "Ubuntu",
    "osType": "Linux",
    "podSubnetId": null,
    "powerState": {
      "code": "Running"
    },
    "provisioningState": "Succeeded",
    "proximityPlacementGroupId": null,
    "resourceGroup": "rg-akscostsaving",
    "scaleDownMode": null,
    "scaleSetEvictionPolicy": null,
    "scaleSetPriority": null,
    "spotMaxPrice": null,
    "tags": null,
    "type": "Microsoft.ContainerService/managedClusters/agentPools",
    "typePropertiesType": "VirtualMachineScaleSets",
    "upgradeSettings": {
      "drainTimeoutInMinutes": null,
      "maxSurge": null
    },
    "vmSize": "Standard_DS2_v2",
    "vnetSubnetId": null,
    "workloadRuntime": null
  }
]

ノード プールのノード数をゼロに増減させる

あとで作ったノードに対して、ノード数の値を 0 に設定します。

Azure Cloud Shell
$ az aks nodepool scale \
    --resource-group $RESOURCE_GROUP \
    --cluster-name $AKS_CLUSTER_NAME \
    --name batchprocpl \
    --node-count 0
{
  "availabilityZones": null,
  "count": 0,
  "creationData": null,
  "currentOrchestratorVersion": "1.27.3",
  "enableAutoScaling": false,  // ここ
  "enableEncryptionAtHost": false,
  "enableFips": false,
  "enableNodePublicIp": false,
  "enableUltraSsd": false,
  "gpuInstanceProfile": null,
  "hostGroupId": null,
  "id": "/subscriptions/xxxxxxxx-xxxxxxxxx-xxxx-xxxxxxxxxxxx/resourcegroups/rg-akscostsaving/providers/Microsoft.ContainerService/managedClusters/akscostsaving-5733/agentPools/batchprocpl",
  "kubeletConfig": null,
  "kubeletDiskType": "OS",
  "linuxOsConfig": null,
  "maxCount": null,
  "maxPods": 110,
  "minCount": null,
  "mode": "User",
  "name": "batchprocpl",
  "nodeImageVersion": "AKSUbuntu-2204gen2containerd-202309.06.0",
  "nodeLabels": null,
  "nodePublicIpPrefixId": null,
  "nodeTaints": null,
  "orchestratorVersion": "1.27.3",
  "osDiskSizeGb": 128,
  "osDiskType": "Managed",
  "osSku": "Ubuntu",
  "osType": "Linux",
  "podSubnetId": null,
  "powerState": {
    "code": "Running"
  },
  "provisioningState": "Succeeded",
  "proximityPlacementGroupId": null,
  "resourceGroup": "rg-akscostsaving",
  "scaleDownMode": "Delete",
  "scaleSetEvictionPolicy": null,
  "scaleSetPriority": null,
  "spotMaxPrice": null,
  "tags": null,
  "type": "Microsoft.ContainerService/managedClusters/agentPools",
  "typePropertiesType": "VirtualMachineScaleSets",
  "upgradeSettings": {
    "drainTimeoutInMinutes": null,
    "maxSurge": null
  },
  "vmSize": "Standard_DS2_v2",
  "vnetSubnetId": null,
  "workloadRuntime": null
}

ノードカウントが0で、"enableAutoScaling": false になっているので、ノードの追加は手動でやる必要がある。

Kubernetes のコンテキストを構成する

  1. Kubernetes クラスターのコンテキストを構成します。kubectlを使うために必要。
Azure Cloud Shell
$ az aks get-credentials \
    --resource-group $RESOURCE_GROUP \
    --name $AKS_CLUSTER_NAME
Merged "akscostsaving-5733" as current context in /home/user/.kube/config
  1. ノード プール内のノードを一覧表示します。
Azure Cloud Shell
$ kubectl get nodes
NAME                                STATUS   ROLES   AGE   VERSION
aks-nodepool1-35854533-vmss000000   Ready    agent   18m   v1.27.3
aks-nodepool1-35854533-vmss000001   Ready    agent   18m   v1.27.3

クラスターで使用可能（自動スケールできるってこと？）なのはbatchprocplノードではなく、nodepool1(先に作った方)なので、ここに出てくるのは先に作ったnodepool1のノードだけになる。

リソースのクリーンアップ

Azure Portalから、作成したリソースグループを削除する。

Kubernetesのコンテキストを削除する。

Azure Cloud Shell
$ kubectl config delete-context akscostsaving-5733

おわりに

むふーむずかしい

Discussion