K8sクラスタのアップグレードでつまづいたこと

v.1.28 => v.1.29のアップグレードでいろいろつまづいた話です。

v.1.29.0が見つからない

kubeadm 1.29.0をインストールしようとしても、Not foundと出てくる

$ sudo apt-get update && sudo apt-get install -y kubeadm='1.29.0'
Hit:2 https://download.docker.com/linux/ubuntu focal InRelease
...
Reading state information... Done
E: Version '1.29.0' for 'kubeadm' was not found

以下でもなぜかv1.28が出てこない。

$ apt list --upgradeable
Listing... Done
containerd.io/focal 1.6.28-2 amd64 [upgradable from: 1.6.6-1]
kubeadm/unknown 1.28.8-1.1 amd64 [upgradable from: 1.28.0-1.1]

解決策は/etc/apt/sources.list.d/kubernetes.listを直さないといけないっぽいです。

$ vim /etc/apt/sources.list.d/kubernetes.list
deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.28/deb/ /
+ deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.29/deb/ /

Reference: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#installing-kubeadm-kubelet-and-kubectl

そしたら出てくるようになりました。

$ sudo apt update
$ sudo apt-cache madison kubeadm
   kubeadm | 1.29.3-1.1 | https://pkgs.k8s.io/core:/stable:/v1.29/deb  Packages
   kubeadm | 1.29.2-1.1 | https://pkgs.k8s.io/core:/stable:/v1.29/deb  Packages
   kubeadm | 1.29.1-1.1 | https://pkgs.k8s.io/core:/stable:/v1.29/deb  Packages
   kubeadm | 1.29.0-1.1 | https://pkgs.k8s.io/core:/stable:/v1.29/deb  Packages

それでもまだKubeadmのインストールに失敗

それでもまだインストールに失敗します。

$ sudo apt-get install -y kubeadm=1.29.0-1.1
...
The following held packages will be changed:
  kubeadm
The following packages will be upgraded:
  kubeadm
1 upgraded, 0 newly installed, 0 to remove and 5 not upgraded.
E: Held packages were changed and -y was used without --allow-change-held-packages.

-yをつけてはいけないようです。それか--allow...オプションをつけるか。

$ sudo apt-get install kubeadm=1.29.0-1.1
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following held packages will be changed:
  kubeadm
The following packages will be upgraded:
  kubeadm
1 upgraded, 0 newly installed, 0 to remove and 5 not upgraded.
Need to get 10.1 MB of archives.
After this operation, 2,511 kB disk space will be freed.
Do you want to continue? [Y/n] Y
Get:1 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.29/deb  kubeadm 1.29.0-1.1 [10.1 MB]
Fetched 10.1 MB in 0s (40.0 MB/s)
debconf: delaying package configuration, since apt-utils is not installed
(Reading database ... 20469 files and directories currently installed.)
Preparing to unpack .../kubeadm_1.29.0-1.1_amd64.deb ...
Unpacking kubeadm (1.29.0-1.1) over (1.28.0-1.1) ...
Setting up kubeadm (1.29.0-1.1) ...

アップグレードするときはバージョンを指定すること

でアップグレードするときもバージョンを指定しないと、1.28.8のままでした

$ kubeadm upgrade plan v1.29.0 # NEED TO SPECIFY VERSION HERE
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks.
[upgrade] Running cluster health checks
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: v1.28.0
[upgrade/versions] kubeadm version: v1.29.0
[upgrade/versions] Target version: v1.29.0
[upgrade/versions] Latest version in the v1.28 series: v1.29.0

Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT   CURRENT       TARGET
kubelet     2 x v1.28.0   v1.29.0

Upgrade to the latest version in the v1.28 series:

COMPONENT                 CURRENT   TARGET
kube-apiserver            v1.28.0   v1.29.0
kube-controller-manager   v1.28.0   v1.29.0
kube-scheduler            v1.28.0   v1.29.0
kube-proxy                v1.28.0   v1.29.0
CoreDNS                   v1.10.1   v1.11.1
etcd                      3.5.9-0   3.5.10-0

You can now apply the upgrade by executing the following command:

        kubeadm upgrade apply v1.29.0

アップグレードをApplyしてようやくうまくいきました

kubeadm upgrade apply v1.29.0
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
...

[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.29.0". Enjoy!

kubeletのアップグレード

ここは比較的シンプルでした。
Controlplaneのアップグレード

$ systemctl daemon-reload

$ systemctl restart kubelet

$ kubectl uncordon controlplane
node/controlplane uncordoned

$ kubectl get node
NAME           STATUS   ROLES           AGE   VERSION
controlplane   Ready    control-plane   63m   v1.29.0
node01         Ready    <none>          61m   v1.28.0

node01のほうもsshしてほぼいっしょです。