🔖

AKS の kube-system で動いてるリソースたち

2022/04/29に公開

kube-system の中身ってこうなってるんだなって話。
よくわからないんだけど ConfigMap が取れてないことがわかったので後半に追加。

$ kubectl get all -n kube-system
NAME                                      READY   STATUS    RESTARTS   AGE
pod/azure-ip-masq-agent-68j88             1/1     Running   0          7d2h
pod/azure-ip-masq-agent-dwjtg             1/1     Running   0          3m43s
pod/azure-ip-masq-agent-t4ddc             1/1     Running   0          7d2h
# daemonSet なので 3 つある
pod/coredns-69c47794-6xnlq                1/1     Running   0          12h
pod/coredns-69c47794-cgn9k                1/1     Running   0          7d2h
# coredns-autoscaler によって設定されているので現状では 2 pods
pod/coredns-autoscaler-5f85dc856b-vbhbb   1/1     Running   0          11d
# coredns の replicas を扱ってる
pod/csi-azuredisk-node-74qr5              3/3     Running   0          7d2h
pod/csi-azuredisk-node-f96pj              3/3     Running   0          3m43s
pod/csi-azuredisk-node-lvc76              3/3     Running   0          7d2h
# Azure Disk を PV として扱うため、たぶん
pod/csi-azurefile-node-2sxj9              3/3     Running   0          3m43s
pod/csi-azurefile-node-gpwv5              3/3     Running   0          10d
pod/csi-azurefile-node-nml4l              3/3     Running   0          10d
# 同じく Azure Files を PV として扱うため
pod/konnectivity-agent-586fcb7848-bzz4s   1/1     Running   0          11d
pod/konnectivity-agent-586fcb7848-d9sss   1/1     Running   0          11d
# Kuberenetes の Konnectivity という機能を使って AKS Control-plane <-> Nodes の間で VPN を張っているらしい。cf.) https://jpaztech.github.io/blog/containers/aks-control-plane-node-communication/
pod/kube-proxy-jct8w                      1/1     Running   0          5m25s
pod/kube-proxy-pjztn                      1/1     Running   0          5m13s
pod/kube-proxy-wf7qw                      1/1     Running   0          3m43s
# 内部的な proxy だった気がする
pod/metrics-server-774f99dbf4-6v9b8       1/1     Running   1          11d
# たぶん Azure Monitor とつながってる

reverse proxy とか load balancer の機能を持っている Service たち。

NAME                     TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)         AGE
service/kube-dns         ClusterIP   10.0.0.10      <none>        53/UDP,53/TCP   11d
# 内部的な DNS サーバ
service/metrics-server   ClusterIP   10.0.131.223   <none>        443/TCP         11d
# たぶんここにメトリックを送ると Azure Monitor の metric として出てくるんだと思う

1 node に 1 つおかれる DaemonSet たち。

NAME                                    DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/azure-ip-masq-agent      3         3         3       3            3           <none>          11d
daemonset.apps/csi-azuredisk-node       3         3         3       3            3           <none>          11d
daemonset.apps/csi-azuredisk-node-win   0         0         0       0            0           <none>          11d
# たぶん Azure Disk を PV として使うためのもの
daemonset.apps/csi-azurefile-node       3         3         3       3            3           <none>          11d
daemonset.apps/csi-azurefile-node-win   0         0         0       0            0           <none>          11d
# 同じく Azure Files のやつだと思う
daemonset.apps/kube-proxy               3         3         3       3            3           <none>          11d
# 内部的な proxy をやってる

pod の上位概念らしい Deployment。

NAME                                 READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/coredns              2/2     2            2           11d
# 内部 DNS サーバ
deployment.apps/coredns-autoscaler   1/1     1            1           11d
# coredns pod の scale を扱ってる。node の数とか、合計 core 数を監視してる
deployment.apps/konnectivity-agent   2/2     2            2           11d
deployment.apps/metrics-server       1/1     1            1           11d
# たぶん Azure Monitor とつながってる

pod の上位概念の一つの ReplicaSet。直接デプロイしたわけじゃないから触らないほうがいいはず、たぶん

NAME                                            DESIRED   CURRENT   READY   AGE
replicaset.apps/coredns-69c47794                2         2         2       7d2h
# coredns のそれ
replicaset.apps/coredns-autoscaler-5f85dc856b   1         1         1       11d
# coredns の autoscaler、replicaSet である意味はあんまりわかってない
replicaset.apps/konnectivity-agent-586fcb7848   2         2         2       11d
replicaset.apps/metrics-server-774f99dbf4       1         1         1       11d
# これもなんで replicaSet なのかわかってない

追加で ConfigMap も。

$ kubectl get configmaps -n kube-system
NAME                                 DATA   AGE
azure-ip-masq-agent-config           1      11d
coredns                              1      11d
coredns-autoscaler                   1      11d
coredns-custom                       0      11d
extension-apiserver-authentication   6      11d
kube-root-ca.crt                     1      11d
overlay-upgrade-data                 4      11d

例えば coredns だとこんな感じ。
ゾーン情報が書いてあある感じっぽい。
AKS private cluter を使ってるので一部置き換えてある。
data: が 1 つしかない。

$ kubectl get configmaps coredns -n kube-system -o yaml
apiVersion: v1
data:
  Corefile: |
    private-cl-aks-xxxxxx-xxxxxxxx.xxxxxxxx-xxxx-xxxx-xxxx-52cd5b9677e5.privatelink.southeastasia.azmk8s.io {
        hosts privatelink.hosts private-cl-aks-xxxxxx-xxxxxxxx.xxxxxxxx-xxxx-xxxx-xxxx-52cd5b9677e5.privatelink.southeastasia.azmk8s.io {
          172.16.10.4 private-cl-aks-xxxxxx-xxxxxxxx.xxxxxxxx-xxxx-xxxx-xxxx-52cd5b9677e5.privatelink.southeastasia.azmk8s.io
          fallthrough
        }
    }
    .:53 {
        errors
        ready
        health
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
        import custom/*.override
    }
    import custom/*.server
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"Corefile":"private-cl-aks-xxxxxx-xxxxxxxx.xxxxxxxx-xxxx-xxxx-xxxx-52cd5b9677e5.privatelink.southeastasia.azmk8s.io {\n    hosts privatelink.hosts private-cl-aks-xxxxxx-xxxxxxxx.xxxxxxxx-xxxx-xxxx-xxxx-52cd5b9677e5.privatelink.southeastasia.azmk8s.io {\n      172.16.10.4 private-cl-aks-xxxxxx-xxxxxxxx.xxxxxxxx-xxxx-xxxx-xxxx-52cd5b9677e5.privatelink.southeastasia.azmk8s.io\n      fallthrough\n    }\n}\n.:53 {\n    errors\n    ready\n    health\n    kubernetes cluster.local in-addr.arpa ip6.arpa {\n      pods insecure\n      fallthrough in-addr.arpa ip6.arpa\n    }\n    prometheus :9153\n    forward . /etc/resolv.conf\n    cache 30\n    loop\n    reload\n    loadbalance\n    import custom/*.override\n}\nimport custom/*.server\n"},"kind":"ConfigMap","metadata":{"annotations":{},"labels":{"addonmanager.kubernetes.io/mode":"Reconcile","k8s-app":"kube-dns","kubernetes.io/cluster-service":"true"},"name":"coredns","namespace":"kube-system"}}
  creationTimestamp: "2022-04-17T06:14:44Z"
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
    k8s-app: kube-dns
    kubernetes.io/cluster-service: "true"
  name: coredns
  namespace: kube-system
  resourceVersion: "2073"
  uid: xxxxxxxx-xxxx-xxxx-xxxx-ab14331457c9

overlay-upgrade-data だと data: が 4 つある。
全体は短いけど。

$ kubectl get configmaps overlay-upgrade-data -n kube-system -o yaml
apiVersion: v1
data:
  ccpEtag: xxxxxxxx-xxxx-xxxx-xxxx-ff6bee335c57
  isSLA: "false"
  lastUpgradeTime: "2022-04-29T02:50:05.004653003Z"
  overlaymgrversionGoal: 'Version: v20220417.220420.2 - Branch: refs/heads/official/v20220417
    - git SHA: 47716d522 - Build date / time: 2022-04-20T12:20:39Z'
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"ccpEtag":"xxxxxxxx-xxxx-xxxx-xxxx-ff6bee335c57","isSLA":"false","lastUpgradeTime":"2022-04-29T02:50:05.004653003Z","overlaymgrversionGoal":"Version: v20220417.220420.2 - Branch: refs/heads/official/v20220417 - git SHA: 47716d522 - Build date / time: 2022-04-20T12:20:39Z"},"kind":"ConfigMap","metadata":{"annotations":{},"labels":{"addonmanager.kubernetes.io/mode":"Reconcile","k8s-app":"overlay-upgrade"},"name":"overlay-upgrade-data","namespace":"kube-system"}}
  creationTimestamp: "2022-04-17T06:14:47Z"
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
    k8s-app: overlay-upgrade
  name: overlay-upgrade-data
  namespace: kube-system
  resourceVersion: "3385444"
  uid: xxxxxxxx-xxxx-xxxx-xxxx-d50bad8a67c2
Microsoft (有志)

Discussion