Zenn
😎

kube-apiserver --anonymous-auth=falseの副作用

2020/12/13に公開
Kubernetes
tech

はじめに

公式ドキュメントCustomizing control plane configuration with kubeadmに従って、設定値をカスタマイズしてクラスターを構築してみました。その際、サンプルのとおりにanonymous-auth: "false"を指定するとkubeadm joinでノードを追加できなくなったので紹介します。

環境

この環境でなくても再現するはずですが、お作法として記載しておきます。

  • GCP
  • Ubuntu 18.04 LTS
  • Kubernetes 1.20.0

コントロールプレーンのセットアップ（anonymous-auth: false）

kubeadm.conf
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: v1.20.0
apiServer:
  extraArgs:
    anonymous-auth: "false"

kubeadm init --config kubeadm.conf

ノードのセットアップ（失敗）

kubeadm join 10.146.0.4:6443 --token 5sqb0r.3wblj3xw8v8lpcgg \
>     --discovery-token-ca-cert-hash sha256:722b9ad59b7b0b1aedf7552bb18b6263002cc8d85031d6930b266ab5291cc925 
[preflight] Running pre-flight checks
error execution phase preflight: couldn't validate the identity of the API Server: Unauthorized
To see the stack trace of this error execute with --v=5 or higher

preflightフェーズでエラーとなります。

コントロールプレーンのセットアップ（anonymous-auth: true）

anonymous-auth: "true"を設定するとともに監査ログの出力を設定します。

kubeadm.conf
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: v1.20.0
apiServer:
  extraArgs:
    anonymous-auth: "true"
    audit-log-path: /var/log/kubernetes/audit.log
    audit-policy-file: /etc/kubernetes/audit-policy.yaml
  extraVolumes:
    - name: audit-policy
      hostPath: /home/naoki/audit-policy.yaml
      mountPath: /etc/kubernetes/audit-policy.yaml
      pathType: File
    - name: audit-log
      hostPath: /var/log/kubernetes/
      mountPath: /var/log/kubernetes/
      pathType: DirectoryOrCreate
/home/naoki/audit-policy.yaml
apiVersion: audit.k8s.io/v1
kind: Policy
omitStages:
  - "RequestReceived"
rules:
  - level: RequestResponse
    users: ["system:anonymous"]
  - level: None

kubeadm init --config kubeadm.conf

ノードのセットアップ（成功）

kubeadm join 10.146.0.4:6443 --token r7bvr8.tmq8d1w0knx5rbc8 \
>     --discovery-token-ca-cert-hash sha256:76de8df5d70bd4de8c1c7bbbbffd6b6a2e8fa917fa41b8c4d61687dcd24d03bc
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

監査ログ

system:anonymousユーザーによりkube-public Namespaceにあるcluster-info ConfigMapをgetしています。anonymous-auth: "false"とするとここで失敗してしまいます。

cat /var/log/kubernetes/audit.log | grep -v readyz | grep -v livez | jq .

{
  "kind": "Event",
  "apiVersion": "audit.k8s.io/v1",
  "level": "RequestResponse",
  "auditID": "7fec1444-fa03-4612-84a7-4328e5df78db",
  "stage": "ResponseComplete",
  "requestURI": "/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s",
  "verb": "get",
  "user": {
    "username": "system:anonymous",
    "groups": [
      "system:unauthenticated"
    ]
  },
  "sourceIPs": [
    "10.146.0.5"
  ],
  "userAgent": "kubeadm/v1.20.0 (linux/amd64) kubernetes/af46c47",
  "objectRef": {
    "resource": "configmaps",
    "namespace": "kube-public",
    "name": "cluster-info",
    "apiVersion": "v1"
  },
  "responseStatus": {
    "metadata": {},
    "code": 200
  },
  "responseObject": {
    "kind": "ConfigMap",
    "apiVersion": "v1",
    "metadata": {
      "name": "cluster-info",
      "namespace": "kube-public",
      "uid": "dedc2864-32b0-490c-966a-e7d12b6c4265",
      "resourceVersion": "380",
      "creationTimestamp": "2020-12-13T10:48:32Z",
      "managedFields": [
        {
          "manager": "kubeadm",
          "operation": "Update",
          "apiVersion": "v1",
          "time": "2020-12-13T10:48:32Z",
          "fieldsType": "FieldsV1",
          "fieldsV1": {
            "f:data": {
              ".": {},
              "f:kubeconfig": {}
            }
          }
        },
        {
          "manager": "kube-controller-manager",
          "operation": "Update",
          "apiVersion": "v1",
          "time": "2020-12-13T10:48:36Z",
          "fieldsType": "FieldsV1",
          "fieldsV1": {
            "f:data": {
              "f:jws-kubeconfig-chfo7u": {}
            }
          }
        }
      ]
    },
    "data": {
      "jws-kubeconfig-chfo7u": "eyJhbGciOiJIUzI1NiIsImtpZCI6ImNoZm83dSJ9..RtmX7dr_7u25fmCCnA1g8QnSsSZZUafXNoPCMKbCozo",
      "kubeconfig": "apiVersion: v1\nclusters:\n- cluster:\n    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM1ekNDQWMrZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJd01USXhNekV3TkRneE5sb1hEVE13TVRJeE1URXdORGd4Tmxvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBS2dmCnRWVUFEcndrdzJOUE9GYytpbGR2KzNuL2FGbzllN1dzVjhJRDZBYnNlVHNaZVlWeHpHYWhzeXR6QUFMbmNSOEsKRHdpMWE0aTMyc0ZIUFdkQStxRm5Rd1JSTE4waEJWcUtadUNQSUk2MVIyU3RVNVJBOThIS2h2UkdINjRYejdJRApuMVhjK0k5TmVkS3V0Qk4vTFh2VHMwS3RvMlBwWGllamZKbVUxNXU0RmRtNFA1UVZXNExFQUEwTDIxTkQvb25RClhxRE9kU010ZHpVSVUxbVBWRHk4aG5KREF2Yml3dXh3M2Y4Y2psNXdJVzNIVWVwbi96dkZhdUxLcEloQms1U2YKNlhuVENpN1VaYkIvWjZZMXNpRkRoaUVvM0pGM3NsRlF5aGNYeVd2cW90dE5PZTV4OGZVYzh3RHJmTlpBdXVVVAo3aEVZWEszQzZuTmRWa05nWWFVQ0F3RUFBYU5DTUVBd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0hRWURWUjBPQkJZRUZQbUV4YTJzby9yNmkyVy9tdXRGQjU5M2ZBQkdNQTBHQ1NxR1NJYjMKRFFFQkN3VUFBNElCQVFCT1pkKzFDdloyaUxjREUzZTdtR3ZJYjdvakxjMDZSOVJjZ0FVWms4QXdLdUNScS9nTgpBSFRkYU56dDVReUlrSlVEUFc0OXhMUlNDNkZOeitFRXV4SG9CVEt4ZXVMUHFRdi9QWTRsUXRtUmZoaXExdUJWCm4zT2srUU5xa1hTVWdpWmNDU05WVmVOUGtQQ0hmWHNQK3BxWmRhZmpIMnQzM2NmZUhWdWF5ZjlUVzVuNWRueEsKOUlLY3FOUndUc2ExL3VKbnYvemQvOEZvS1U2eFdzRnMybVBOelRqcUV3Z290YUxiMkdkSkpmaGNDL3JlYmZENgp4NjVnRUpLSWRvWG9Zd2VsTWFGaXc1Q3FiakxsZy9iakVVTEZtWGNyNTY1cGFRMHdBbG1oS0xSd0V6cTFLM1JDCjF4NzFiWU5KOGJDZWlDUmhTTmZwaDUrdzBhY3Z3bFVXOThUSQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==\n    server: https://10.146.0.4:6443\n  name: \"\"\ncontexts: null\ncurrent-context: \"\"\nkind: Config\npreferences: {}\nusers: null\n"
    }
  },
  "requestReceivedTimestamp": "2020-12-13T10:48:45.085069Z",
  "stageTimestamp": "2020-12-13T10:48:45.090189Z",
  "annotations": {
    "authorization.k8s.io/decision": "allow",
    "authorization.k8s.io/reason": "RBAC: allowed by RoleBinding \"kubeadm:bootstrap-signer-clusterinfo/kube-public\" of Role \"kubeadm:bootstrap-signer-clusterinfo\" to User \"system:anonymous\""
  }
}
{
  "kind": "Event",
  "apiVersion": "audit.k8s.io/v1",
  "level": "RequestResponse",
  "auditID": "af15c748-2db3-4e83-9785-1cc3e2b23286",
  "stage": "ResponseComplete",
  "requestURI": "/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s",
  "verb": "get",
  "user": {
    "username": "system:anonymous",
    "groups": [
      "system:unauthenticated"
    ]
  },
  "sourceIPs": [
    "10.146.0.5"
  ],
  "userAgent": "kubeadm/v1.20.0 (linux/amd64) kubernetes/af46c47",
  "objectRef": {
    "resource": "configmaps",
    "namespace": "kube-public",
    "name": "cluster-info",
    "apiVersion": "v1"
  },
  "responseStatus": {
    "metadata": {},
    "code": 200
  },
  "responseObject": {
    "kind": "ConfigMap",
    "apiVersion": "v1",
    "metadata": {
      "name": "cluster-info",
      "namespace": "kube-public",
      "uid": "dedc2864-32b0-490c-966a-e7d12b6c4265",
      "resourceVersion": "380",
      "creationTimestamp": "2020-12-13T10:48:32Z",
      "managedFields": [
        {
          "manager": "kubeadm",
          "operation": "Update",
          "apiVersion": "v1",
          "time": "2020-12-13T10:48:32Z",
          "fieldsType": "FieldsV1",
          "fieldsV1": {
            "f:data": {
              ".": {},
              "f:kubeconfig": {}
            }
          }
        },
        {
          "manager": "kube-controller-manager",
          "operation": "Update",
          "apiVersion": "v1",
          "time": "2020-12-13T10:48:36Z",
          "fieldsType": "FieldsV1",
          "fieldsV1": {
            "f:data": {
              "f:jws-kubeconfig-chfo7u": {}
            }
          }
        }
      ]
    },
    "data": {
      "jws-kubeconfig-chfo7u": "eyJhbGciOiJIUzI1NiIsImtpZCI6ImNoZm83dSJ9..RtmX7dr_7u25fmCCnA1g8QnSsSZZUafXNoPCMKbCozo",
      "kubeconfig": "apiVersion: v1\nclusters:\n- cluster:\n    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM1ekNDQWMrZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJd01USXhNekV3TkRneE5sb1hEVE13TVRJeE1URXdORGd4Tmxvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBS2dmCnRWVUFEcndrdzJOUE9GYytpbGR2KzNuL2FGbzllN1dzVjhJRDZBYnNlVHNaZVlWeHpHYWhzeXR6QUFMbmNSOEsKRHdpMWE0aTMyc0ZIUFdkQStxRm5Rd1JSTE4waEJWcUtadUNQSUk2MVIyU3RVNVJBOThIS2h2UkdINjRYejdJRApuMVhjK0k5TmVkS3V0Qk4vTFh2VHMwS3RvMlBwWGllamZKbVUxNXU0RmRtNFA1UVZXNExFQUEwTDIxTkQvb25RClhxRE9kU010ZHpVSVUxbVBWRHk4aG5KREF2Yml3dXh3M2Y4Y2psNXdJVzNIVWVwbi96dkZhdUxLcEloQms1U2YKNlhuVENpN1VaYkIvWjZZMXNpRkRoaUVvM0pGM3NsRlF5aGNYeVd2cW90dE5PZTV4OGZVYzh3RHJmTlpBdXVVVAo3aEVZWEszQzZuTmRWa05nWWFVQ0F3RUFBYU5DTUVBd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0hRWURWUjBPQkJZRUZQbUV4YTJzby9yNmkyVy9tdXRGQjU5M2ZBQkdNQTBHQ1NxR1NJYjMKRFFFQkN3VUFBNElCQVFCT1pkKzFDdloyaUxjREUzZTdtR3ZJYjdvakxjMDZSOVJjZ0FVWms4QXdLdUNScS9nTgpBSFRkYU56dDVReUlrSlVEUFc0OXhMUlNDNkZOeitFRXV4SG9CVEt4ZXVMUHFRdi9QWTRsUXRtUmZoaXExdUJWCm4zT2srUU5xa1hTVWdpWmNDU05WVmVOUGtQQ0hmWHNQK3BxWmRhZmpIMnQzM2NmZUhWdWF5ZjlUVzVuNWRueEsKOUlLY3FOUndUc2ExL3VKbnYvemQvOEZvS1U2eFdzRnMybVBOelRqcUV3Z290YUxiMkdkSkpmaGNDL3JlYmZENgp4NjVnRUpLSWRvWG9Zd2VsTWFGaXc1Q3FiakxsZy9iakVVTEZtWGNyNTY1cGFRMHdBbG1oS0xSd0V6cTFLM1JDCjF4NzFiWU5KOGJDZWlDUmhTTmZwaDUrdzBhY3Z3bFVXOThUSQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==\n    server: https://10.146.0.4:6443\n  name: \"\"\ncontexts: null\ncurrent-context: \"\"\nkind: Config\npreferences: {}\nusers: null\n"
    }
  },
  "requestReceivedTimestamp": "2020-12-13T10:48:45.098993Z",
  "stageTimestamp": "2020-12-13T10:48:45.101181Z",
  "annotations": {
    "authorization.k8s.io/decision": "allow",
    "authorization.k8s.io/reason": "RBAC: allowed by RoleBinding \"kubeadm:bootstrap-signer-clusterinfo/kube-public\" of Role \"kubeadm:bootstrap-signer-clusterinfo\" to User \"system:anonymous\""
  }
}

おわりに

CIS Kubernetes Benchmarkでもanonymous-auth: "false"とすることを推奨していますが、実際にそんなことはできるのでしょうか。少なくともkubeadmを利用してクラスター構築する場合にはできないことがわかりました。オンプレKubernetes勢はどのようにノードを追加しているのか気になります。

GitHubで編集を提案

Discussion