Zenn
Closed18

kubeadm で Kubernetes Upgrade v1.20 to v1.24

Kubernetes
kubeadm
t_umet_ume
  • 開発用のk8sのバージョンが古いので検証も兼ねてアップグレードしていく。
  • バージョンがだいぶ古いのとアップグレードがバージョンをスキップすることが出来ないので、段階を踏んでアップグレードを行う。
  • kubeadmで構築しているので、kubeadm upgradeでアップグレードさせる。
  • 開発機なので乗っているコンテナの稼働（移動や停止）は気にしない。
  • アップグレード影響は都度確認してみる。
  • クラスター構成
    • OS：Ubuntu
    • ControlPlane：3台
    • Worker：3台
  • スタート：v1.20.6
  • ゴール：v.1.24.x
t_umet_ume

手順は公式を参考に。
https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/

  • ControlPlane(Master)⇒　Workerの順にアップグレードする。
  • 基本的な流れはバージョン通して以下の流れ
  1. 対象バージョン選定
  2. kubeadmアップグレード
  3. アップグレードプランの確認（影響確認など）
  4. kubeadm upgrade（ControlPlaneを1台ずつ）
  5. kubelet/kubectl アップグレードと kubelet 再起動(ControlPlaneを1台ずつ)
  6. Workerに対して、上記2・4・5を1台ずつ実行する
t_umet_ume

1.20 ⇒ 1.21

  • 1.21の新しいバージョンを確認する
$ apt update
・・・
64 packages can be upgraded. Run 'apt list --upgradable' to see them.

$ apt-cache madison kubeadm | grep 1.21
   kubeadm | 1.21.14-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
・・・
   kubeadm |  1.21.0-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
  • ControlPlaneからアップグレードする
  • まずは1台目
#kubeadmをアップグレード 
$ apt-mark unhold kubeadm && \
apt-get update && apt-get install -y kubeadm=1.21.14-00 && \
apt-mark hold kubeadm

・・・
The following packages were automatically installed and are no longer required:
  linux-headers-4.15.0-176 linux-headers-4.15.0-176-generic linux-image-4.15.0-176-generic 
・・・
2 upgraded, 0 newly installed, 0 to remove and 62 not upgraded.
・・・
Setting up kubeadm (1.21.14-00) ...

# バージョン確認
$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.14", GitCommit:"0f77da5bd4809927e15d1658fb4aa8f13ad890a5", GitTreeState:"clean", BuildDate:"2022-06-15T14:16:13Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}

# アップグレードプランの確認
$ kubeadm upgrade plan
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks.
[upgrade] Running cluster health checks
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: v1.20.6
[upgrade/versions] kubeadm version: v1.21.14
I0916 18:18:20.886443   25444 version.go:254] remote version is much newer: v1.25.1; falling back to: stable-1.21
[upgrade/versions] Target version: v1.21.14
[upgrade/versions] Latest version in the v1.20 series: v1.20.15

W0916 18:18:21.777037   25444 kubelet.go:275] The 'cgroupDriver' value in the KubeletConfiguration is empty. Starting from 1.22, 'kubeadm upgrade' will default an empty value to the 'systemd' cgroup driver. The cgroup driver between the container runtime and the kubelet must match! To learn more about this see: https://kubernetes.io/docs/setup/production-environment/container-runtimes/
Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT   CURRENT       TARGET
kubelet     6 x v1.20.6   v1.20.15

Upgrade to the latest version in the v1.20 series:

COMPONENT                 CURRENT    TARGET
kube-apiserver            v1.20.6    v1.20.15
kube-controller-manager   v1.20.6    v1.20.15
kube-scheduler            v1.20.6    v1.20.15
kube-proxy                v1.20.6    v1.20.15
CoreDNS                   1.7.0      v1.8.0
etcd                      3.4.13-0   3.4.13-0

You can now apply the upgrade by executing the following command:

        kubeadm upgrade apply v1.20.15

_____________________________________________________________________

Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT   CURRENT       TARGET
kubelet     6 x v1.20.6   v1.21.14

Upgrade to the latest stable version:

COMPONENT                 CURRENT    TARGET
kube-apiserver            v1.20.6    v1.21.14
kube-controller-manager   v1.20.6    v1.21.14
kube-scheduler            v1.20.6    v1.21.14
kube-proxy                v1.20.6    v1.21.14
CoreDNS                   1.7.0      v1.8.0
etcd                      3.4.13-0   3.4.13-0

You can now apply the upgrade by executing the following command:

        kubeadm upgrade apply v1.21.14

_____________________________________________________________________


The table below shows the current state of component configs as understood by this version of kubeadm.
Configs that have a "yes" mark in the "MANUAL UPGRADE REQUIRED" column require manual config upgrade or
resetting to kubeadm defaults before a successful upgrade can be performed. The version to manually
upgrade to is denoted in the "PREFERRED VERSION" column.

API GROUP                 CURRENT VERSION   PREFERRED VERSION   MANUAL UPGRADE REQUIRED
kubeproxy.config.k8s.io   v1alpha1          v1alpha1            no
kubelet.config.k8s.io     v1beta1           v1beta1             no
_____________________________________________________________________
  • 1台目のアップグレード
# Upgrade
$ sudo kubeadm upgrade apply v1.21.14
・・・
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.21.14". Enjoy!
[upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.

# ノードの確認、1台目なのでまだUpgradeされていない？
$ kubectl get node -owide
NAME   STATUS   ROLES                  AGE    VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION       CONTAINER-RUNTIME
cp01   Ready    control-plane,master   498d   v1.20.6   192.168.10.61   <none>        Ubuntu 18.04.5 LTS   4.15.0-192-generic   containerd://1.4.4
cp02   Ready    control-plane,master   498d   v1.20.6   192.168.10.62   <none>        Ubuntu 18.04.5 LTS   4.15.0-192-generic   containerd://1.4.4
cp03   Ready    control-plane,master   498d   v1.20.6   192.168.10.63   <none>        Ubuntu 18.04.5 LTS   4.15.0-192-generic   containerd://1.4.4
nd01   Ready    <none>                 498d   v1.20.6   192.168.10.71   <none>        Ubuntu 18.04.5 LTS   4.15.0-192-generic   containerd://1.4.4
nd02   Ready    <none>                 498d   v1.20.6   192.168.10.72   <none>        Ubuntu 18.04.5 LTS   4.15.0-192-generic   containerd://1.4.4
nd03   Ready    <none>                 498d   v1.20.6   192.168.10.73   <none>        Ubuntu 18.04.5 LTS   4.15.0-192-generic   containerd://1.4.4

# kubectlで見ても1.20
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.6", GitCommit:"8a62859e515889f07e3e3be6a1080413f17cf2c3", GitTreeState:"clean", BuildDate:"2021-04-15T03:28:42Z", GoVersion:"go1.15.10", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.6", GitCommit:"8a62859e515889f07e3e3be6a1080413f17cf2c3", GitTreeState:"clean", BuildDate:"2021-04-15T03:19:55Z", GoVersion:"go1.15.10", Compiler:"gc", Platform:"linux/amd64"}

まずは1台目成功

t_umet_ume
  • ２台目、３台目を実施
# 2台目、kubeadmをアップグレード
$ apt-mark unhold kubeadm && \
apt-get update && apt-get install -y kubeadm=1.21.14-00 && \
apt-mark hold kubeadm

# 2台目をアップグレード（１台目とコマンドが異なる）
$ sudo kubeadm upgrade node
・・・
[upgrade] The control plane instance for this node was successfully updated!
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[upgrade] The configuration for this node was successfully updated!
[upgrade] Now you should go ahead and upgrade the kubelet package using your package manager.

# ３台目も同様にアップグレード
# 作業後のノードの状態は以下の通り。まだバージョンは上がっていなかった。
$ kubectl get node -owide
NAME   STATUS   ROLES                  AGE    VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION       CONTAINER-RUNTIME
cp01   Ready    control-plane,master   498d   v1.20.6   192.168.10.61   <none>        Ubuntu 18.04.5 LTS   4.15.0-192-generic   containerd://1.4.4
cp02   Ready    control-plane,master   498d   v1.20.6   192.168.10.62   <none>        Ubuntu 18.04.5 LTS   4.15.0-192-generic   containerd://1.4.4
cp03   Ready    control-plane,master   498d   v1.20.6   192.168.10.63   <none>        Ubuntu 18.04.5 LTS   4.15.0-192-generic   containerd://1.4.4
nd01   Ready    <none>                 498d   v1.20.6   192.168.10.71   <none>        Ubuntu 18.04.5 LTS   4.15.0-192-generic   containerd://1.4.4
nd02   Ready    <none>                 498d   v1.20.6   192.168.10.72   <none>        Ubuntu 18.04.5 LTS   4.15.0-192-generic   containerd://1.4.4
nd03   Ready    <none>                 498d   v1.20.6   192.168.10.73   <none>        Ubuntu 18.04.5 LTS   4.15.0-192-generic   containerd://1.4.4
t_umet_ume
  • ControlPlaneの作業の最後に、各ControlPlaneのkubectlとkubeletをアップグレードする。
$ apt-get update && \
  apt-get install -y --allow-change-held-packages kubelet=1.21.14-00 kubectl=1.21.14-00
・・・
Setting up kubelet (1.21.14-00) ...
Setting up kubectl (1.21.14-00) ...

# この時点でバージョンが変更されていた。kubeletのバージョン？
$ kubectl get node -owide
NAME   STATUS   ROLES                  AGE    VERSION    INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION       CONTAINER-RUNTIME
cp01   Ready    control-plane,master   498d   v1.21.14   192.168.10.61   <none>        Ubuntu 18.04.5 LTS   4.15.0-192-generic   containerd://1.4.4
cp02   Ready    control-plane,master   498d   v1.20.6    192.168.10.62   <none>        Ubuntu 18.04.5 LTS   4.15.0-192-generic   containerd://1.4.4
cp03   Ready    control-plane,master   498d   v1.20.6    192.168.10.63   <none>        Ubuntu 18.04.5 LTS   4.15.0-192-generic   containerd://1.4.4
nd01   Ready    <none>                 498d   v1.20.6    192.168.10.71   <none>        Ubuntu 18.04.5 LTS   4.15.0-192-generic   containerd://1.4.4
nd02   Ready    <none>                 498d   v1.20.6    192.168.10.72   <none>        Ubuntu 18.04.5 LTS   4.15.0-192-generic   containerd://1.4.4
nd03   Ready    <none>                 498d   v1.20.6    192.168.10.73   <none>        Ubuntu 18.04.5 LTS   4.15.0-192-generic   containerd://1.4.4

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.6", GitCommit:"8a62859e515889f07e3e3be6a1080413f17cf2c3", GitTreeState:"clean", BuildDate:"2021-04-15T03:28:42Z", GoVersion:"go1.15.10", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.14", GitCommit:"0f77da5bd4809927e15d1658fb4aa8f13ad890a5", GitTreeState:"clean", BuildDate:"2022-06-15T14:11:36Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}

# kubelet再起動
$ sudo systemctl daemon-reload
$ sudo systemctl restart kubelet

# 同様の作業を２台目・３台目にも行う
# 以下、全台実施後の状態
$ kubectl get node -owide
NAME   STATUS   ROLES                  AGE    VERSION    INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION       CONTAINER-RUNTIME
cp01   Ready    control-plane,master   498d   v1.21.14   192.168.10.61   <none>        Ubuntu 18.04.5 LTS   4.15.0-192-generic   containerd://1.4.4
cp02   Ready    control-plane,master   498d   v1.21.14   192.168.10.62   <none>        Ubuntu 18.04.5 LTS   4.15.0-192-generic   containerd://1.4.4
cp03   Ready    control-plane,master   498d   v1.21.14   192.168.10.63   <none>        Ubuntu 18.04.5 LTS   4.15.0-192-generic   containerd://1.4.4
nd01   Ready    <none>                 498d   v1.20.6    192.168.10.71   <none>        Ubuntu 18.04.5 LTS   4.15.0-192-generic   containerd://1.4.4
nd02   Ready    <none>                 498d   v1.20.6    192.168.10.72   <none>        Ubuntu 18.04.5 LTS   4.15.0-192-generic   containerd://1.4.4
nd03   Ready    <none>                 498d   v1.20.6    192.168.10.73   <none>        Ubuntu 18.04.5 LTS   4.15.0-192-generic   containerd://1.4.4

ControlPlane完了

t_umet_ume
  • Worker Nodeを実施していく
  • 流れはControlPlaneと同様
  1. kubeadm コマンドをアップグレード
  2. Workerをアップグレード
  3. kubelet/kubectl のアップグレード
  4. kubelet 再起動
  • ノードの台数分以下を実施していく
# Upgrade kubeadm 
# apt-getのバージョンによってはシンプルにコマンドが実行できる
$ apt-get update && \
apt-get install -y --allow-change-held-packages kubeadm=1.21.14-00

# Upgrade node
$ sudo kubeadm upgrade node
・・・
[upgrade] The configuration for this node was successfully updated!
[upgrade] Now you should go ahead and upgrade the kubelet package using your package manager.

# Upgrade kubelet/kubectl
$ apt-get update && \
  apt-get install -y --allow-change-held-packages kubelet=1.21.14-00 kubectl=1.21.14-00

# Restart kubelet
$ sudo systemctl daemon-reload
$ sudo systemctl restart kubelet
t_umet_ume
  • 1.20 ⇒ 1.21のアップグレード完了
# nodeの確認
$ kubectl get node -owide
NAME   STATUS   ROLES                  AGE    VERSION    INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION       CONTAINER-RUNTIME
cp01   Ready    control-plane,master   498d   v1.21.14   192.168.10.61   <none>        Ubuntu 18.04.5 LTS   4.15.0-192-generic   containerd://1.4.4
cp02   Ready    control-plane,master   498d   v1.21.14   192.168.10.62   <none>        Ubuntu 18.04.5 LTS   4.15.0-192-generic   containerd://1.4.4
cp03   Ready    control-plane,master   498d   v1.21.14   192.168.10.63   <none>        Ubuntu 18.04.5 LTS   4.15.0-192-generic   containerd://1.4.4
nd01   Ready    <none>                 498d   v1.21.14   192.168.10.71   <none>        Ubuntu 18.04.5 LTS   4.15.0-192-generic   containerd://1.4.4
nd02   Ready    <none>                 498d   v1.21.14   192.168.10.72   <none>        Ubuntu 18.04.5 LTS   4.15.0-192-generic   containerd://1.4.4
nd03   Ready    <none>                 498d   v1.21.14   192.168.10.73   <none>        Ubuntu 18.04.5 LTS   4.15.0-192-generic   containerd://1.4.4

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.14", GitCommit:"0f77da5bd4809927e15d1658fb4aa8f13ad890a5", GitTreeState:"clean", BuildDate:"2022-06-15T14:17:29Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.14", GitCommit:"0f77da5bd4809927e15d1658fb4aa8f13ad890a5", GitTreeState:"clean", BuildDate:"2022-06-15T14:11:36Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}
  • 本番ではdrainをして順番に実施していくと安全
  • 雑にアップグレードをかけたが Readnessが少し切れるくらいで、コンテナ再作成はなさそうに見えた
    • 起動していたコンテナ：rook-ceph / harbor / nginx-ingress / concourse / cert-manager
  • 公式にもある通り、アップグレードで証明書も更新されていた。
$ kubeadm certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Sep 16, 2023 18:25 UTC   364d            ca                      no
apiserver                  Sep 16, 2023 18:25 UTC   364d            ca                      no
apiserver-etcd-client      Sep 16, 2023 18:25 UTC   364d            etcd-ca                 no
apiserver-kubelet-client   Sep 16, 2023 18:25 UTC   364d            ca                      no
controller-manager.conf    Sep 16, 2023 18:25 UTC   364d            ca                      no
etcd-healthcheck-client    Sep 16, 2023 18:24 UTC   364d            etcd-ca                 no
etcd-peer                  Sep 16, 2023 18:24 UTC   364d            etcd-ca                 no
etcd-server                Sep 16, 2023 18:24 UTC   364d            etcd-ca                 no
front-proxy-client         Sep 16, 2023 18:25 UTC   364d            front-proxy-ca          no
scheduler.conf             Sep 16, 2023 18:25 UTC   364d            ca                      no

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      May 04, 2031 16:31 UTC   8y              no
etcd-ca                 May 04, 2031 16:31 UTC   8y              no
front-proxy-ca          May 04, 2031 16:31 UTC   8y              no
t_umet_ume

1.21 ⇒ 1.22

  • 基本的な流れはコマンドは一緒なので作業ログを一部割愛する。
最新バージョン確認
$ apt-cache madison kubeadm | grep 1.22
   kubeadm | 1.22.14-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
・・・
   kubeadm |  1.22.0-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
  • 1.21.14 ⇒ 1.22.14 にアップグレードする。
t_umet_ume
  • kubeadmをアップグレード後、kubeadm upgrade plan でアップグレードプランを確認する
  • クラスタ的には影響はなさそう？
$ kubeadm upgrade plan
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks.
[upgrade] Running cluster health checks
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: v1.21.14
[upgrade/versions] kubeadm version: v1.22.14
I0917 04:06:39.967925    3897 version.go:255] remote version is much newer: v1.25.1; falling back to: stable-1.22
[upgrade/versions] Target version: v1.22.14
[upgrade/versions] Latest version in the v1.21 series: v1.21.14

Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT   CURRENT        TARGET
kubelet     6 x v1.21.14   v1.22.14

Upgrade to the latest stable version:

COMPONENT                 CURRENT    TARGET
kube-apiserver            v1.21.14   v1.22.14
kube-controller-manager   v1.21.14   v1.22.14
kube-scheduler            v1.21.14   v1.22.14
kube-proxy                v1.21.14   v1.22.14
CoreDNS                   v1.8.0     v1.8.4
etcd                      3.4.13-0   3.5.0-0

You can now apply the upgrade by executing the following command:

        kubeadm upgrade apply v1.22.14

_____________________________________________________________________


The table below shows the current state of component configs as understood by this version of kubeadm.
Configs that have a "yes" mark in the "MANUAL UPGRADE REQUIRED" column require manual config upgrade or
resetting to kubeadm defaults before a successful upgrade can be performed. The version to manually
upgrade to is denoted in the "PREFERRED VERSION" column.

API GROUP                 CURRENT VERSION   PREFERRED VERSION   MANUAL UPGRADE REQUIRED
kubeproxy.config.k8s.io   v1alpha1          v1alpha1            no
kubelet.config.k8s.io     v1beta1           v1beta1             no
_____________________________________________________________________
t_umet_ume
  • ControlPlaneでkubeadm upgrade後に一部コンテナ起動出来なくなった
  • 利用していたcert-manager（0.14）の一部Podがサポート対象外となった模様
  • 1.22で廃止されたAPIに影響した？

https://cert-manager.io/docs/installation/supported-releases/
https://qiita.com/tkusumi/items/2e3c69849ce2a4b50a90

  • バージョンアップが必要までは分かったので、レプリカ0にして一旦保留
  • kubelet アップグレード、再起動時にkube-systemの一部PodがErrorに一時的になったが、自動で起動し直されて正常稼働に戻った。
    • api-serverのPodがErrorになっていた。
  • Workerアップグレード時にPodの再作成が行われていたが、自動で基本的には復旧
t_umet_ume
  • 1.21 ⇒ 1.22 完了
  • 手順は1.20と同じ方法でアップグレード出来た。
  • 1.22のAPI廃止の影響があるため、古いマニフェストを使っていると動作に影響が出そう。
  • 最終的な結果は以下の通り
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.14", GitCommit:"bccf857df03c5a99a35e34020b3b63055f0c12ec", GitTreeState:"clean", BuildDate:"2022-09-14T22:41:51Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.14", GitCommit:"bccf857df03c5a99a35e34020b3b63055f0c12ec", GitTreeState:"clean", BuildDate:"2022-09-14T22:36:04Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}

$ kubectl get node -owide
NAME   STATUS   ROLES                  AGE    VERSION    INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION       CONTAINER-RUNTIME
cp01   Ready    control-plane,master   498d   v1.22.14   192.168.10.61   <none>        Ubuntu 18.04.5 LTS   4.15.0-192-generic   containerd://1.4.4
cp02   Ready    control-plane,master   498d   v1.22.14   192.168.10.62   <none>        Ubuntu 18.04.5 LTS   4.15.0-192-generic   containerd://1.4.4
cp03   Ready    control-plane,master   498d   v1.22.14   192.168.10.63   <none>        Ubuntu 18.04.5 LTS   4.15.0-192-generic   containerd://1.4.4
nd01   Ready    <none>                 498d   v1.22.14   192.168.10.71   <none>        Ubuntu 18.04.5 LTS   4.15.0-192-generic   containerd://1.4.4
nd02   Ready    <none>                 498d   v1.22.14   192.168.10.72   <none>        Ubuntu 18.04.5 LTS   4.15.0-192-generic   containerd://1.4.4
nd03   Ready    <none>                 498d   v1.22.14   192.168.10.73   <none>        Ubuntu 18.04.5 LTS   4.15.0-192-generic   containerd://1.4.4
t_umet_ume

1.22 ⇒ 1.23

上述同様にまずはバージョンの選定から。

バージョン選定
$ apt-cache madison kubeadm | grep 1.23
   kubeadm | 1.23.11-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
・・・
   kubeadm |  1.23.0-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
  • 1.22.14 から 1.23.11 にアップグレードする。
t_umet_ume
  • 手順は今までと一緒。
  • アップグレードプランで確認。
  • 警告等はなさそう。
$ kubeadm upgrade plan
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks.
[upgrade] Running cluster health checks
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: v1.22.14
[upgrade/versions] kubeadm version: v1.23.11
I0917 13:34:59.412271   26674 version.go:255] remote version is much newer: v1.25.1; falling back to: stable-1.23
[upgrade/versions] Target version: v1.23.11
[upgrade/versions] Latest version in the v1.22 series: v1.22.14

Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT   CURRENT        TARGET
kubelet     6 x v1.22.14   v1.23.11

Upgrade to the latest stable version:

COMPONENT                 CURRENT    TARGET
kube-apiserver            v1.22.14   v1.23.11
kube-controller-manager   v1.22.14   v1.23.11
kube-scheduler            v1.22.14   v1.23.11
kube-proxy                v1.22.14   v1.23.11
CoreDNS                   v1.8.4     v1.8.6
etcd                      3.5.0-0    3.5.1-0

You can now apply the upgrade by executing the following command:

        kubeadm upgrade apply v1.23.11

_____________________________________________________________________


The table below shows the current state of component configs as understood by this version of kubeadm.
Configs that have a "yes" mark in the "MANUAL UPGRADE REQUIRED" column require manual config upgrade or
resetting to kubeadm defaults before a successful upgrade can be performed. The version to manually
upgrade to is denoted in the "PREFERRED VERSION" column.

API GROUP                 CURRENT VERSION   PREFERRED VERSION   MANUAL UPGRADE REQUIRED
kubeproxy.config.k8s.io   v1alpha1          v1alpha1            no
kubelet.config.k8s.io     v1beta1           v1beta1             no
_____________________________________________________________________
t_umet_ume
  • 1.22 ⇒ 1.23 完了
  • 手順は同じ方法でアップグレード出来た。
  • 1.22と違い、1.23のアップグレードで影響のあったPodはなかった。
  • 最終的な結果は以下の通り
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.11", GitCommit:"dc2898b20c6bd9602ae1c3b51333e2e4640ed249", GitTreeState:"clean", BuildDate:"2022-09-14T16:40:13Z", GoVersion:"go1.17.13", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.11", GitCommit:"dc2898b20c6bd9602ae1c3b51333e2e4640ed249", GitTreeState:"clean", BuildDate:"2022-09-14T16:32:41Z", GoVersion:"go1.17.13", Compiler:"gc", Platform:"linux/amd64"}

$ kubectl get node -owide
NAME   STATUS   ROLES                  AGE    VERSION    INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION       CONTAINER-RUNTIME
cp01   Ready    control-plane,master   499d   v1.23.11   192.168.10.61   <none>        Ubuntu 18.04.5 LTS   4.15.0-192-generic   containerd://1.4.4
cp02   Ready    control-plane,master   499d   v1.23.11   192.168.10.62   <none>        Ubuntu 18.04.5 LTS   4.15.0-192-generic   containerd://1.4.4
cp03   Ready    control-plane,master   499d   v1.23.11   192.168.10.63   <none>        Ubuntu 18.04.5 LTS   4.15.0-192-generic   containerd://1.4.4
nd01   Ready    <none>                 499d   v1.23.11   192.168.10.71   <none>        Ubuntu 18.04.5 LTS   4.15.0-192-generic   containerd://1.4.4
nd02   Ready    <none>                 499d   v1.23.11   192.168.10.72   <none>        Ubuntu 18.04.5 LTS   4.15.0-192-generic   containerd://1.4.4
nd03   Ready    <none>                 499d   v1.23.11   192.168.10.73   <none>        Ubuntu 18.04.5 LTS   4.15.0-192-generic   containerd://1.4.4
t_umet_ume

1.23 ⇒ 1.24

  • 今回最後のアップグレード
  • 今まで通りの手法でアップグレードしていきます。
バージョン確認
$ apt-cache madison kubeadm | grep 1.24
   kubeadm |  1.24.5-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
・・・
   kubeadm |  1.24.0-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
  • 1.23.11 から 1.24.5 にアップグレードする。
t_umet_ume
  • アップグレードプランを確認
$ kubeadm upgrade plan
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
W0917 17:17:01.310117   31318 initconfiguration.go:120] Usage of CRI endpoints without URL scheme is deprecated and can cause kubelet errors in the future. Automatically prepending scheme "unix" to the "criSocket" with value "/run/containerd/containerd.sock". Please update your configuration!
[preflight] Running pre-flight checks.
[upgrade] Running cluster health checks
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: v1.23.11
[upgrade/versions] kubeadm version: v1.24.5
I0917 17:17:08.804625   31318 version.go:255] remote version is much newer: v1.25.1; falling back to: stable-1.24
[upgrade/versions] Target version: v1.24.5
[upgrade/versions] Latest version in the v1.23 series: v1.23.11

Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT   CURRENT        TARGET
kubelet     6 x v1.23.11   v1.24.5

Upgrade to the latest stable version:

COMPONENT                 CURRENT    TARGET
kube-apiserver            v1.23.11   v1.24.5
kube-controller-manager   v1.23.11   v1.24.5
kube-scheduler            v1.23.11   v1.24.5
kube-proxy                v1.23.11   v1.24.5
CoreDNS                   v1.8.6     v1.8.6
etcd                      3.5.1-0    3.5.3-0

You can now apply the upgrade by executing the following command:

        kubeadm upgrade apply v1.24.5

_____________________________________________________________________


The table below shows the current state of component configs as understood by this version of kubeadm.
Configs that have a "yes" mark in the "MANUAL UPGRADE REQUIRED" column require manual config upgrade or
resetting to kubeadm defaults before a successful upgrade can be performed. The version to manually
upgrade to is denoted in the "PREFERRED VERSION" column.

API GROUP                 CURRENT VERSION   PREFERRED VERSION   MANUAL UPGRADE REQUIRED
kubeproxy.config.k8s.io   v1alpha1          v1alpha1            no
kubelet.config.k8s.io     v1beta1           v1beta1             no
_____________________________________________________________________
t_umet_ume
  • 1.23 ⇒ 1.24 完了
  • 手順は今までと同じ方法でアップグレード出来た。
  • 1.23と同様に、1.24のアップグレードも影響のあったPodはなかった。
  • 最終的な結果は以下の通り
$ kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.5", GitCommit:"e979822c185a14537054f15808a118d7fcce1d6e", GitTreeState:"clean", BuildDate:"2022-09-14T16:42:36Z", GoVersion:"go1.18.6", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.4
Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.5", GitCommit:"e979822c185a14537054f15808a118d7fcce1d6e", GitTreeState:"clean", BuildDate:"2022-09-14T16:35:41Z", GoVersion:"go1.18.6", Compiler:"gc", Platform:"linux/amd64"}

$ kubectl get node -owide
NAME   STATUS   ROLES           AGE    VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION       CONTAINER-RUNTIME
cp01   Ready    control-plane   499d   v1.24.5   192.168.10.61   <none>        Ubuntu 18.04.5 LTS   4.15.0-192-generic   containerd://1.4.4
cp02   Ready    control-plane   499d   v1.24.5   192.168.10.62   <none>        Ubuntu 18.04.5 LTS   4.15.0-192-generic   containerd://1.4.4
cp03   Ready    control-plane   499d   v1.24.5   192.168.10.63   <none>        Ubuntu 18.04.5 LTS   4.15.0-192-generic   containerd://1.4.4
nd01   Ready    <none>          499d   v1.24.5   192.168.10.71   <none>        Ubuntu 18.04.5 LTS   4.15.0-192-generic   containerd://1.4.4
nd02   Ready    <none>          499d   v1.24.5   192.168.10.72   <none>        Ubuntu 18.04.5 LTS   4.15.0-192-generic   containerd://1.4.4
nd03   Ready    <none>          499d   v1.24.5   192.168.10.73   <none>        Ubuntu 18.04.5 LTS   4.15.0-192-generic   containerd://1.4.4
t_umet_ume
  • 手順自体は同じ事の繰り返しだったので、意外と簡単に行うことができた。
  • 動いていたPodに一部影響があったので、バージョンアップ時には変更点（主にマニフェスト周辺）には注意を配ることが大切
このスクラップは2022/10/19にクローズされました