Zenn
Closed11

kubeadmでetcd分離型クラスタを構築するぞ

Kubernetes
kubeadm
RyuSARyuSA

まずは準備から
https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/setup-ha-etcd-with-kubeadm/
↑に書いてある通り、まずはkubeadmとkubeletをインストール。それとDockerも必要……と言われているけど、Dockerはいらない子なのでCRIコンポーネントとしてcontainerdで代用します

RyuSARyuSA

Dockerはいらない子なのでCRIコンポーネントとしてcontainerdで代用します

お前あとで痛い目見るから覚悟しろよ？(悲しみ)

RyuSARyuSA

はいキマシタワー、dockerからcontainerdに変更してこけるポイント
https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/setup-ha-etcd-with-kubeadm/#setting-up-the-cluster

CRIをインストールしてkubeletの設定を変更して再起動したところ、kubeletがrunningになってくれない。ひどい
うーん……kubeadmがCRIを探してきてくれないのか、kubeletがインストールされているCRIを自動で取り込んでくれずにdockerを探しに行ってしまった。

Dec 21 14:06:50 etcd03 kubelet[6956]: F1221 14:06:50.396893    6956 server.go:269] failed to run Kubelet: failed to get docker version: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

なのでちょっと修正。
https://github.com/kata-containers/documentation/blob/master/how-to/how-to-use-k8s-with-cri-containerd-and-kata.md#configure-kubelet-to-use-containerd
KaraContainerのリポジトリだけどkubeletがcontainerdを使うように設定する方法が書いてあった。ので一旦これをそのまま採用。(--container-runtime=remoteとはどういった意味なんだろう？要調査)

root@etcd03:~# cat << EOF > /etc/systemd/system/kubelet.service.d/20-etcd-service-manager.conf
[Service]
ExecStart=
ExecStart=/usr/bin/kubelet --address=127.0.0.1 --pod-manifest-path=/etc/kubernetes/manifests --cgroup-driver=systemd --container-runtime=remote --runtime-request-timeout=15m --container-runtime-endpoint=unix:///run/containerd/containerd.sock
Restart=always
EOF
root@etcd03:~# systemctl daemon-reload
root@etcd03:~# systemctl restart kubelet

これでOKかな？(Argumentで指定するなWarnが死ぬほど出てて申し訳ない)

RyuSARyuSA

--container-runtime=remoteとはどういった意味なんだろう？

気になりすぎて夜も眠れたのでちょっと調査
https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/

--container-runtime string Default: docker
The container runtime to use. Possible values: docker, remote.

とのこと。つまり……dockershimを使うかCRIで通信をするかって選択するところかなるほど。

RyuSARyuSA

証明書を作成する部分

root@etcd01:~# kubeadm init phase certs etcd-server --config=/tmp/${HOST2}/kubeadmcfg.yaml
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [etcd01 localhost] and IPs [10.32.0.61 127.0.0.1 ::1 10.32.0.52]
root@etcd01:~# kubeadm init phase certs etcd-peer --config=/tmp/${HOST2}/kubeadmcfg.yaml
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [etcd01 localhost] and IPs [10.32.0.61 127.0.0.1 ::1 10.32.0.52]
root@etcd01:~# kubeadm init phase certs etcd-healthcheck-client --config=/tmp/${HOST2}/kubeadmcfg.yaml
[certs] Generating "etcd/healthcheck-client" certificate and key
root@etcd01:~# kubeadm init phase certs apiserver-etcd-client --config=/tmp/${HOST2}/kubeadmcfg.yaml
[certs] Generating "apiserver-etcd-client" certificate and key
root@etcd01:~# cp -R /etc/kubernetes/pki /tmp/${HOST2}/
root@etcd01:~# ll /tmp/${HOST2}/
total 16
drwxr-xr-x  3 root root 4096 Dec 27 06:45 ./
drwxrwxrwt 14 root root 4096 Dec 27 06:36 ../
-rw-r--r--  1 root root  613 Dec 27 06:07 kubeadmcfg.yaml
drwxr-xr-x  3 root root 4096 Dec 27 06:45 pki/
root@etcd01:~# ll /tmp/${HOST2}/pki/
total 20
drwxr-xr-x 3 root root 4096 Dec 27 06:45 ./
drwxr-xr-x 3 root root 4096 Dec 27 06:45 ../
-rw-r--r-- 1 root root 1135 Dec 27 06:45 apiserver-etcd-client.crt
-rw------- 1 root root 1675 Dec 27 06:45 apiserver-etcd-client.key
drwxr-xr-x 2 root root 4096 Dec 27 06:45 etcd/
root@etcd01:~# ll /tmp/${HOST2}/pki/etcd
total 40
drwxr-xr-x 2 root root 4096 Dec 27 06:45 ./
drwxr-xr-x 3 root root 4096 Dec 27 06:45 ../
-rw-r--r-- 1 root root 1058 Dec 27 06:45 ca.crt
-rw------- 1 root root 1675 Dec 27 06:45 ca.key
-rw-r--r-- 1 root root 1139 Dec 27 06:45 healthcheck-client.crt
-rw------- 1 root root 1679 Dec 27 06:45 healthcheck-client.key
-rw-r--r-- 1 root root 1180 Dec 27 06:45 peer.crt
-rw------- 1 root root 1675 Dec 27 06:45 peer.key
-rw-r--r-- 1 root root 1180 Dec 27 06:45 server.crt
-rw------- 1 root root 1679 Dec 27 06:45 server.key

うーーーん？？一旦これらは何をしているのだろうか……ちょっと証明書の中身を覗いてみる。

root@etcd01:~# openssl x509 -in /tmp/${HOST2}/pki/etcd/healthcheck-client.crt -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 2387255149029865085 (0x21213be8ea25b67d)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = etcd-ca
        Validity
            Not Before: Dec 27 06:08:06 2020 GMT
            Not After : Dec 27 06:45:32 2021 GMT
        Subject: O = system:masters, CN = kube-etcd-healthcheck-client
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus:
                    (snip)
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Client Authentication
            X509v3 Authority Key Identifier:
                keyid:8A:07:9A:39:98:D1:B4:AB:38:BF:79:00:6C:E7:76:B6:D8:18:53:7E
    Signature Algorithm: sha256WithRSAEncryption
         (snip)
root@etcd01:~# openssl x509 -in /tmp/${HOST2}/pki/etcd/peer.crt -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 1061505567252827896 (0xebb39b077344ef8)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = etcd-ca
        Validity
            Not Before: Dec 27 06:08:06 2020 GMT
            Not After : Dec 27 06:45:26 2021 GMT
        Subject: CN = etcd01
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus:
                    (snip)
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Authority Key Identifier:
                keyid:8A:07:9A:39:98:D1:B4:AB:38:BF:79:00:6C:E7:76:B6:D8:18:53:7E
            X509v3 Subject Alternative Name:
                DNS:etcd01, DNS:localhost, IP Address:10.32.0.61, IP Address:127.0.0.1, IP Address:0:0:0:0:0:0:0:1, IP Address:10.32.0.52
    Signature Algorithm: sha256WithRSAEncryption
         (snip)
root@etcd01:~# openssl x509 -in /tmp/${HOST2}/pki/etcd/server.crt -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 8384364539923290651 (0x745b3f2075825a1b)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = etcd-ca
        Validity
            Not Before: Dec 27 06:08:06 2020 GMT
            Not After : Dec 27 06:45:19 2021 GMT
        Subject: CN = etcd01
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus:
                    (snip)
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Authority Key Identifier:
                keyid:8A:07:9A:39:98:D1:B4:AB:38:BF:79:00:6C:E7:76:B6:D8:18:53:7E
            X509v3 Subject Alternative Name:
                DNS:etcd01, DNS:localhost, IP Address:10.32.0.61, IP Address:127.0.0.1, IP Address:0:0:0:0:0:0:0:1, IP Address:10.32.0.52
    Signature Algorithm: sha256WithRSAEncryption
         (snip)

とりあえずコマンドで作成されたCRTのIssuerがすべて"etcd-ca"らしいことはわかった。このetcd-caって誰なんだろう……？

と思って、最初に作成した/etc/kubernetes/pki/etcd/ca.crtを確認してみる。

root@etcd01:~# openssl x509 -in /etc/kubernetes/pki/etcd/ca.crt -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 0 (0x0)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = etcd-ca
        Validity
            Not Before: Dec 27 06:08:06 2020 GMT
            Not After : Dec 25 06:08:06 2030 GMT
        Subject: CN = etcd-ca
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus:
                    (snip)
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment, Certificate Sign
            X509v3 Basic Constraints: critical
                CA:TRUE
            X509v3 Subject Key Identifier:
                8A:07:9A:39:98:D1:B4:AB:38:BF:79:00:6C:E7:76:B6:D8:18:53:7E
    Signature Algorithm: sha256WithRSAEncryption
         (snip)

見つけた！etcd-caというCA局はどうやらオレオレ証明書で運用しているらしい( ..)φ
この証明書は誰が管理してくれるのだろうか……一応CA局の証明書は10年間、その他の証明書は1年間は保証されているっぽいけど……(あとで調べる)

次に進む前にそれぞれの鍵の利用用途を調べてみよう。

RyuSARyuSA

ここに証明書の用途が書いてある。

https://kubernetes.io/docs/setup/best-practices/certificates/#certificate-paths

で、今回利用作成されたCRTについては……etcdとetcdctlのドキュメントに利用用途とオプションが書いてる

https://etcd.io/docs/v3.2.17/op-guide/security/

https://github.com/etcd-io/etcd/tree/master/etcdctl

  • ca
    • CA証明書
    • etcdctlを叩くときに利用する
  • kube-etcd-healthcheck-client
    • 証明書情報
    • etcdctlを叩くときに利用する
  • kube-etcd-peer
    • etcd間でクラスタリングを行うための証明書
  • server
    • etcdへ接続する際に利用する証明書

おぉ～、なるほどね……名前のまんまかw
ところでkube-etcd-healthcheck-clientはetcdctlを叩くときに使うらしいけど、どんなコマンドを叩いているのだろうか……どこかで調べてみよう

RyuSARyuSA

ファイルを転送した後にetcdを起動するところ

root@etcd02:~# kubeadm init phase etcd local --config=/home/ubuntu/kubeadmcfg.yaml
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
root@etcd02:~# ll /etc/kubernetes/manifests
total 12
drwxr-xr-x 2 root root 4096 Dec 27 07:47 ./
drwxr-xr-x 4 root root 4096 Dec 27 07:45 ../
-rw------- 1 root root 2244 Dec 27 07:47 etcd.yaml
root@etcd02:~# cat /etc/kubernetes/manifests/etcd.yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubeadm.kubernetes.io/etcd.advertise-client-urls: https://10.32.0.212:2379
  creationTimestamp: null
  labels:
    component: etcd
    tier: control-plane
  name: etcd
  namespace: kube-system
spec:
  containers:
  - command:
    - etcd
    - --advertise-client-urls=https://10.32.0.212:2379
    - --cert-file=/etc/kubernetes/pki/etcd/server.crt
    - --client-cert-auth=true
    - --data-dir=/var/lib/etcd
    - --initial-advertise-peer-urls=https://10.32.0.212:2380
    - --initial-cluster=infra0=https://10.32.0.61:2380,infra1=https://10.32.0.212:2380,infra2=https://10.32.0.52:2380
    - --initial-cluster-state=new
    - --key-file=/etc/kubernetes/pki/etcd/server.key
    - --listen-client-urls=https://10.32.0.212:2379
    - --listen-metrics-urls=http://127.0.0.1:2381
    - --listen-peer-urls=https://10.32.0.212:2380
    - --name=infra1
    - --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt
    - --peer-client-cert-auth=true
    - --peer-key-file=/etc/kubernetes/pki/etcd/peer.key
    - --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
    - --snapshot-count=10000
    - --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
    image: k8s.gcr.io/etcd:3.4.13-0
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 8
      httpGet:
        host: 127.0.0.1
        path: /health
        port: 2381
        scheme: HTTP
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 15
    name: etcd
    resources:
      requests:
        cpu: 100m
        ephemeral-storage: 100Mi
        memory: 100Mi
    startupProbe:
      failureThreshold: 24
      httpGet:
        host: 127.0.0.1
        path: /health
        port: 2381
        scheme: HTTP
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 15
    volumeMounts:
    - mountPath: /var/lib/etcd
      name: etcd-data
    - mountPath: /etc/kubernetes/pki/etcd
      name: etcd-certs
  hostNetwork: true
  priorityClassName: system-node-critical
  volumes:
  - hostPath:
      path: /etc/kubernetes/pki/etcd
      type: DirectoryOrCreate
    name: etcd-certs
  - hostPath:
      path: /var/lib/etcd
      type: DirectoryOrCreate
    name: etcd-data

ほう、etcdはStaticPodで起動しているのか。そして/var/lib/etcdにetcdのデータが保存されているとのこと。

で、今回はDockerではなくcontainerdで構築したので公式のDockerコマンドが使えない。(起動しているかな？w)
マウントのオプションとかいろいろ調べる時間もあれだったので、すでに起動しているPodの中でetcdctlを叩くことにする。

root@etcd02:~# crictl --runtime-endpoint unix:///run/containerd/containerd.sock exec -it 7783ae39d648d etcdctl \
> --cert /etc/kubernetes/pki/etcd/peer.crt \
> --key /etc/kubernetes/pki/etcd/peer.key \
> --cacert /etc/kubernetes/pki/etcd/ca.crt \
> --endpoints https://10.32.0.61:2379 \
> endpoint health --cluster
https://10.32.0.212:2379 is healthy: successfully committed proposal: took = 12.702899ms
https://10.32.0.52:2379 is healthy: successfully committed proposal: took = 12.556305ms
https://10.32.0.61:2379 is healthy: successfully committed proposal: took = 15.65244ms

ヨシ、起動している！！etcdクラスタ構築完了！

RyuSARyuSA

masterノード(候補)にアクセスして、ca.crtとapiserver-etcd-client.crtとapiserver-etcd-client.keyを/etc/kubernetes/pkiのそれっぽいところにコピーしてくる。

あとは適切なkubeadmin-config.yamlを作成して

apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: stable
controlPlaneEndpoint: "適当なNLB:6443"
etcd:
    external:
        endpoints:
        - https://10.32.0.61:2379
        - https://10.32.0.212:2379
        - https://10.32.0.52:2379
        caFile: /etc/kubernetes/pki/etcd/ca.crt
        certFile: /etc/kubernetes/pki/apiserver-etcd-client.crt
        keyFile: /etc/kubernetes/pki/apiserver-etcd-client.key

kubeadmin initを叩くだけ！

root@master01:~# kubeadm init --config kubeadm-config.yaml --upload-certs
[init] Using Kubernetes version: v1.20.1
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local master01 ha-apiserver.elb.ap-northeast-1.amazonaws.com] and IPs [10.96.0.1 10.32]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] External etcd mode: Skipping etcd/ca certificate authority generation
[certs] External etcd mode: Skipping etcd/server certificate generation
[certs] External etcd mode: Skipping etcd/peer certificate generation
[certs] External etcd mode: Skipping etcd/healthcheck-client certificate generation
[certs] External etcd mode: Skipping apiserver-etcd-client certificate generation
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 14.508139 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.20" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[upload-certs] Using certificate key:
d26af10381b36715efdd227421748a454f2fdb50ba9bebdde8a5f9a1b5efa7e6
[mark-control-plane] Marking the node master01 as control-plane by adding the labels "node-role.kubernetes.io/master=''" and "node-role.kubernetes.io/control-plane='' (deprecated)"
[mark-control-plane] Marking the node master01 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: z54gox.suml2a96k49apuxo
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!
(snip)
root@master01:~# export KUBECONFIG=/etc/kubernetes/admin.conf
root@master01:~# kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"
serviceaccount/weave-net created
clusterrole.rbac.authorization.k8s.io/weave-net created
clusterrolebinding.rbac.authorization.k8s.io/weave-net created
role.rbac.authorization.k8s.io/weave-net created
rolebinding.rbac.authorization.k8s.io/weave-net created
daemonset.apps/weave-net created
root@master01:~# kubectl get pods --all-namespaces
NAMESPACE     NAME                               READY   STATUS    RESTARTS   AGE
kube-system   coredns-74ff55c5b-88nc8            1/1     Running   0          2m28s
kube-system   coredns-74ff55c5b-hq8j5            1/1     Running   0          2m28s
kube-system   kube-apiserver-master01            1/1     Running   0          2m27s
kube-system   kube-controller-manager-master01   1/1     Running   0          2m27s
kube-system   kube-proxy-9b5pk                   1/1     Running   0          2m28s
kube-system   kube-scheduler-master01            1/1     Running   0          2m27s
kube-system   weave-net-l25kl                    2/2     Running   0          28s
root@master01:~# kubectl get nodes
NAME       STATUS   ROLES                  AGE     VERSION
master01   Ready    control-plane,master   2m48s   v1.20.1

ｳｨｯｽOK完了した！

このスクラップは2020/12/27にクローズされました