🦔
Kubernetes v1.25.3 から v1.26.1 に kubeadm upgrade したら containerd が古くて失敗
Upgrading kubeadm clusters に従って作業したメモ。
OS は Ubuntu 22.04 でした
imksoo@k8smaster:~$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.1 LTS"
containerd は 1.5.9 でした
imksoo@k8smaster:~$ apt show containerd
Package: containerd
Version: 1.5.9-0ubuntu3.1
Built-Using: golang-1.18 (= 1.18.1-1ubuntu1)
Priority: optional
Section: admin
Origin: Ubuntu
Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
Original-Maintainer: Debian Go Packaging Team <pkg-go-maintainers@lists.alioth.debian.org>
Bugs: https://bugs.launchpad.net/ubuntu/+filebug
Installed-Size: 110 MB
Depends: runc (>= 1.0.0~rc2~), libc6 (>= 2.34)
Breaks: docker.io (<< 19.03.13-0ubuntu4)
Homepage: https://containerd.io
Download-Size: 28.1 MB
APT-Manual-Installed: yes
APT-Sources: http://www.ftp.ne.jp/Linux/packages/ubuntu/archive jammy-updates/main amd64 Packages
Description: daemon to control runC
Containerd is a daemon to control runC, built for performance and density.
Containerd leverages runC's advanced features such as seccomp and user
namespace support as well as checkpoint and restore for cloning and live
migration of containers.
.
This package contains the binaries.
N: There is 1 additional record. Please use the '-a' switch to see it
imksoo@k8smaster:~$
kubeadm のバージョンを確認する
imksoo@k8smaster:~$ kubectl version -o json
{
"clientVersion": {
"major": "1",
"minor": "25",
"gitVersion": "v1.25.3",
"gitCommit": "434bfd82814af038ad94d62ebe59b133fcb50506",
"gitTreeState": "clean",
"buildDate": "2022-10-12T10:57:26Z",
"goVersion": "go1.19.2",
"compiler": "gc",
"platform": "linux/amd64"
},
"kustomizeVersion": "v4.5.7",
"serverVersion": {
"major": "1",
"minor": "25",
"gitVersion": "v1.25.3",
"gitCommit": "434bfd82814af038ad94d62ebe59b133fcb50506",
"gitTreeState": "clean",
"buildDate": "2022-10-12T10:49:09Z",
"goVersion": "go1.19.2",
"compiler": "gc",
"platform": "linux/amd64"
}
}
imksoo@k8smaster:~$ kubeadm version -o json
{
"clientVersion": {
"major": "1",
"minor": "25",
"gitVersion": "v1.25.3",
"gitCommit": "434bfd82814af038ad94d62ebe59b133fcb50506",
"gitTreeState": "clean",
"buildDate": "2022-10-12T10:55:36Z",
"goVersion": "go1.19.2",
"compiler": "gc",
"platform": "linux/amd64"
}
}
imksoo@k8smaster:~$ apt update
imksoo@k8smaster:~$ apt-cache madison kubeadm | grep 1.26
kubeadm | 1.26.1-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
kubeadm | 1.26.0-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
imksoo@k8smaster:~$
kubeadm をバージョンアップする
imksoo@k8smaster:~$ sudo apt install kubeadm
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages will be upgraded:
kubeadm
1 upgraded, 0 newly installed, 0 to remove and 6 not upgraded.
Need to get 9,732 kB of archives.
After this operation, 2,961 kB of additional disk space will be used.
Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.26.1-00 [9,732 kB]
Fetched 9,732 kB in 1s (14.6 MB/s)
(Reading database ... 109876 files and directories currently installed.)
Preparing to unpack .../kubeadm_1.26.1-00_amd64.deb ...
Unpacking kubeadm (1.26.1-00) over (1.25.3-00) ...
Setting up kubeadm (1.26.1-00) ...
Scanning processes...
Scanning candidates...
Scanning linux images...
Restarting services...
systemctl restart containerd.service
Service restarts being deferred:
systemctl restart networkd-dispatcher.service
systemctl restart systemd-logind.service
systemctl restart unattended-upgrades.service
systemctl restart user@1000.service
No containers need to be restarted.
No user sessions are running outdated binaries.
No VM guests are running outdated hypervisor (qemu) binaries on this host.
imksoo@k8smaster:~$
imksoo@k8smaster:~$ sudo apt-mark hold kubeadm
kubeadm set on hold.
kubeadm upgrade plan を見る
imksoo@k8smaster:~$ sudo kubeadm upgrade plan
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks.
[upgrade] Running cluster health checks
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: v1.25.3
[upgrade/versions] kubeadm version: v1.26.1
[upgrade/versions] Target version: v1.26.1
[upgrade/versions] Latest version in the v1.25 series: v1.25.6
W0129 22:26:42.170540 735482 configset.go:177] error unmarshaling configuration schema.GroupVersionKind{Group:"kubeproxy.config.k8s.io", Version:"v1alpha1", Kind:"KubeProxyConfiguration"}: strict decoding error: unknown field "udpIdleTimeout"
Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT CURRENT TARGET
kubelet 2 x v1.25.3 v1.25.6
1 x v1.25.4 v1.25.6
Upgrade to the latest version in the v1.25 series:
COMPONENT CURRENT TARGET
kube-apiserver v1.25.3 v1.25.6
kube-controller-manager v1.25.3 v1.25.6
kube-scheduler v1.25.3 v1.25.6
kube-proxy v1.25.3 v1.25.6
CoreDNS v1.9.3 v1.9.3
etcd 3.5.4-0 3.5.6-0
You can now apply the upgrade by executing the following command:
kubeadm upgrade apply v1.25.6
_____________________________________________________________________
Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT CURRENT TARGET
kubelet 2 x v1.25.3 v1.26.1
1 x v1.25.4 v1.26.1
Upgrade to the latest stable version:
COMPONENT CURRENT TARGET
kube-apiserver v1.25.3 v1.26.1
kube-controller-manager v1.25.3 v1.26.1
kube-scheduler v1.25.3 v1.26.1
kube-proxy v1.25.3 v1.26.1
CoreDNS v1.9.3 v1.9.3
etcd 3.5.4-0 3.5.6-0
You can now apply the upgrade by executing the following command:
kubeadm upgrade apply v1.26.1
_____________________________________________________________________
The table below shows the current state of component configs as understood by this version of kubeadm.
Configs that have a "yes" mark in the "MANUAL UPGRADE REQUIRED" column require manual config upgrade or
resetting to kubeadm defaults before a successful upgrade can be performed. The version to manually
upgrade to is denoted in the "PREFERRED VERSION" column.
API GROUP CURRENT VERSION PREFERRED VERSION MANUAL UPGRADE REQUIRED
kubeproxy.config.k8s.io v1alpha1 v1alpha1 no
kubelet.config.k8s.io v1beta1 v1beta1 no
_____________________________________________________________________
imksoo@k8smaster:~$
kubeadm upgrade apply v1.26.1 を実行した
imksoo@k8smaster:~$ sudo kubeadm upgrade apply v1.26.1
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
W0129 22:37:02.646316 739050 configset.go:177] error unmarshaling configuration schema.GroupVersionKind{Group:"kubeproxy.config.k8s.io", Version:"v1alpha1", Kind:"KubeProxyConfiguration"}: strict decoding error: unknown field "udpIdleTimeout"
[preflight] Running pre-flight checks.
[upgrade] Running cluster health checks
[upgrade/version] You have chosen to change the cluster version to "v1.26.1"
[upgrade/versions] Cluster version: v1.25.3
[upgrade/versions] kubeadm version: v1.26.1
[upgrade] Are you sure you want to proceed? [y/N]: y
[upgrade/prepull] Pulling images required for setting up a Kubernetes cluster
[upgrade/prepull] This might take a minute or two, depending on the speed of your internet connection
[upgrade/prepull] You can also perform this action in beforehand using 'kubeadm config images pull'
[preflight] Some fatal errors occurred:
[ERROR ImagePull]: failed to pull image registry.k8s.io/kube-apiserver:v1.26.1: output: time="2023-01-29T22:37:15+09:00" level=fatal msg="validate service connection: CRI v1 image API is not implemented for endpoint \"unix:///run/containerd/containerd.sock\": rpc error: code = Unimplemented desc = unknown service runtime.v1.ImageService"
, error: exit status 1
[ERROR ImagePull]: failed to pull image registry.k8s.io/kube-controller-manager:v1.26.1: output: time="2023-01-29T22:37:15+09:00" level=fatal msg="validate service connection: CRI v1 image API is not implemented for endpoint \"unix:///run/containerd/containerd.sock\": rpc error: code = Unimplemented desc = unknown service runtime.v1.ImageService"
, error: exit status 1
[ERROR ImagePull]: failed to pull image registry.k8s.io/kube-scheduler:v1.26.1: output: time="2023-01-29T22:37:16+09:00" level=fatal msg="validate service connection: CRI v1 image API is not implemented for endpoint \"unix:///run/containerd/containerd.sock\": rpc error: code = Unimplemented desc = unknown service runtime.v1.ImageService"
, error: exit status 1
[ERROR ImagePull]: failed to pull image registry.k8s.io/kube-proxy:v1.26.1: output: time="2023-01-29T22:37:16+09:00" level=fatal msg="validate service connection: CRI v1 image API is not implemented for endpoint \"unix:///run/containerd/containerd.sock\": rpc error: code = Unimplemented desc = unknown service runtime.v1.ImageService"
, error: exit status 1
[ERROR ImagePull]: failed to pull image registry.k8s.io/pause:3.9: output: time="2023-01-29T22:37:16+09:00" level=fatal msg="validate service connection: CRI v1 image API is not implemented for endpoint \"unix:///run/containerd/containerd.sock\": rpc error: code = Unimplemented desc = unknown service runtime.v1.ImageService"
, error: exit status 1
[ERROR ImagePull]: failed to pull image registry.k8s.io/etcd:3.5.6-0: output: time="2023-01-29T22:37:16+09:00" level=fatal msg="validate service connection: CRI v1 image API is not implemented for endpoint \"unix:///run/containerd/containerd.sock\": rpc error: code = Unimplemented desc = unknown service runtime.v1.ImageService"
, error: exit status 1
[ERROR ImagePull]: failed to pull image registry.k8s.io/coredns/coredns:v1.9.3: output: time="2023-01-29T22:37:16+09:00" level=fatal msg="validate service connection: CRI v1 image API is not implemented for endpoint \"unix:///run/containerd/containerd.sock\": rpc error: code = Unimplemented desc = unknown service runtime.v1.ImageService"
, error: exit status 1
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher
imksoo@k8smaster:~$
CRI v1 image API is not implemented for endpoint は containerd のバージョンが古いときに出るエラーなので、containerdのバージョンアップを先にする必要がありそう。
ちょっと今日はここで中断して方策を考えることにします。
Discussion