Zenn
Open14

proxmox上でkubeadmを使ったk8sクラスタを構築する

ほなふくほなふく

環境

Proxmox上にubuntu 22.04のコンテナを建ててそこに構築していく。

k8s version

export VERSION=1.27

コンテナランタイムのインストール

最近話題らしいのでCRI-Oを試してみる
https://github.com/cri-o/cri-o/blob/main/install.md#apt-based-operating-systems

OS=xUbuntu_22.04
echo $OS
# libseccomp2をインストールする
# debian10以下ではリポジトリの追加が必要らしいが、ubuntu22ではaptでlibseccomp2 2.5.3-2ubuntu2が降ってくるので良さそう
# 私の環境では元から入っていた
apt-get update && apt-get upgrade -y
apt install -y libseccomp2

# cri-oリポジトリの追加
echo "deb [signed-by=/usr/share/keyrings/libcontainers-archive-keyring.gpg] https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/ /" > /etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list
echo "deb [signed-by=/usr/share/keyrings/libcontainers-crio-archive-keyring.gpg] https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable:/cri-o:/$VERSION/$OS/ /" > /etc/apt/sources.list.d/devel:kubic:libcontainers:stable:cri-o:$VERSION.list

# リポジトリのkeyを追加
apt-get install -y gpg curl
curl -L https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/Release.key | gpg --dearmor -o /usr/share/keyrings/libcontainers-archive-keyring.gpg
curl -L https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable:/cri-o:/$VERSION/$OS/Release.key | gpg --dearmor -o /usr/share/keyrings/libcontainers-crio-archive-keyring.gpg

# cri-o cri-o-runcのインストール
apt-get update
apt-get install cri-o cri-o-runc

ドキュメントによると/etc/crio/crio.conf.d/01-crio-runc.confにあるランタイムルートを編集してくれとのことだったが、確認すると既に以下の内容が記録されていたため何もしなかった

[crio.runtime.runtimes.runc]
runtime_path = "/usr/lib/cri-o-runc/sbin/runc"
runtime_type = "oci"
runtime_root = "/run/runc"
ほなふくほなふく

kubeのツール群をインストール

https://kubernetes.io/ja/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#kubeadm-kubelet-kubectlのインストール

apt-get update
apt-get install -y apt-transport-https ca-certificates curl

# gcloudの公開鍵を追加
curl -fsSLo /etc/apt/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg

# kubeのリポジトリを追加
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list

# インストール
apt-get update
apt-get install -y kubelet kubeadm kubectl
apt-mark hold kubelet kubeadm kubectl
ほなふくほなふく

kubeletのcgroupドライバー設定

/etc/default/kubeletの値を変更する必要があるかも、と思ったが最後の方で

CRI-Oやcontainerdといった他のコンテナランタイムのcgroup driverは実行中に自動的に検出されます。

と書いているので何もしなかった

ほなふくほなふく

Cilium CLIのインストール

CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/master/stable.txt)
CLI_ARCH=amd64
if [ "$(uname -m)" = "aarch64" ]; then CLI_ARCH=arm64; fi
curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
sha256sum --check cilium-linux-${CLI_ARCH}.tar.gz.sha256sum
sudo tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin
rm cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}

Ciliumのインストール

kubectlで設定されているクラスタに対しCiliumがインストールされるらしい

cilium install

実行したところ

❌ Cluster name "" is not valid, must match regular expression: ^[a-z0-9]([-a-z0-9]*[a-z0-9])$
↩️ Rolling back installation...

Error: Unable to install Cilium: invalid cluster name

と怒られたので先にkubeadm initしたほうが良さそう

ほなふくほなふく

kubeadm init

最初はパラメーターを絞って実行して後から増やしていく

kubeadm init --control-plane-endpoint 192.168.30.10
エラーコード
[init] Using Kubernetes version: v1.27.1
[preflight] Running pre-flight checks
[preflight] The system verification failed. Printing the output from the verification:
KERNEL_VERSION: 5.15.102-1-pve
OS: Linux
CGROUPS_CPU: enabled
CGROUPS_CPUSET: enabled
CGROUPS_DEVICES: enabled
CGROUPS_FREEZER: enabled
CGROUPS_MEMORY: enabled
CGROUPS_PIDS: enabled
CGROUPS_HUGETLB: enabled
CGROUPS_IO: enabled
error execution phase preflight: [preflight] Some fatal errors occurred:
        [ERROR CRI]: container runtime is not running: output: E0505 05:48:42.542263   16559 remote_runtime.go:616] "Status from runtime service failed" err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial unix /var/run/containerd/containerd.sock: connect: no such file or directory\""
time="2023-05-05T05:48:42Z" level=fatal msg="getting status of runtime: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial unix /var/run/containerd/containerd.sock: connect: no such file or directory\""
, error: exit status 1
        [ERROR SystemVerification]: failed to parse kernel config: unable to load kernel module: "configs", output: "modprobe: FATAL: Module configs not found in directory /lib/modules/5.15.102-1-pve\n", err: exit status 1
        [ERROR FileContent--proc-sys-net-ipv4-ip_forward]: /proc/sys/net/ipv4/ip_forward contents are not set to 1
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher

コンテナランタイムを明示的に指定してみる

https://kubernetes.io/ja/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#installing-runtime
に書かれているCRI-Oのパスを指定する

kubeadm init --control-plane-endpoint 192.168.30.10 --cri-socket /var/run/crio/crio.sock
エラーコード
W0505 05:57:28.929776   17368 initconfiguration.go:120] Usage of CRI endpoints without URL scheme is deprecated and can cause kubelet errors in the future. Automatically prepending scheme "unix" to the "criSocket" with value "/var/run/crio/crio.sock". Please update your configuration!
[init] Using Kubernetes version: v1.27.1
[preflight] Running pre-flight checks
[preflight] The system verification failed. Printing the output from the verification:
KERNEL_VERSION: 5.15.102-1-pve
OS: Linux
CGROUPS_CPU: enabled
CGROUPS_CPUSET: enabled
CGROUPS_DEVICES: enabled
CGROUPS_FREEZER: enabled
CGROUPS_MEMORY: enabled
CGROUPS_PIDS: enabled
CGROUPS_HUGETLB: enabled
CGROUPS_IO: enabled
error execution phase preflight: [preflight] Some fatal errors occurred:
        [ERROR CRI]: container runtime is not running: output: E0505 05:57:29.581220   17385 remote_runtime.go:616] "Status from runtime service failed" err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial unix /var/run/crio/crio.sock: connect: no such file or directory\""
time="2023-05-05T05:57:29Z" level=fatal msg="getting status of runtime: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial unix /var/run/crio/crio.sock: connect: no such file or directory\""
, error: exit status 1
        [ERROR SystemVerification]: failed to parse kernel config: unable to load kernel module: "configs", output: "modprobe: FATAL: Module configs not found in directory /lib/modules/5.15.102-1-pve\n", err: exit status 1
        [ERROR FileContent--proc-sys-net-ipv4-ip_forward]: /proc/sys/net/ipv4/ip_forward contents are not set to 1
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher

ランタイムパスをruncにする

エラーではno such file or directoryと言われている。
CRI-Oのセットアップ時にランタイムパスでruncを指定しているのでこれを渡してみる。
lsで対象が存在することは確認している

root@asparagus:~# ls -la /usr/lib/cri-o-runc/sbin/runc
-rwxr-xr-x 1 root root 9833960 May 27  2022 /usr/lib/cri-o-runc/sbin/runc

kubeadm init --control-plane-endpoint 192.168.30.10 --cri-socket /usr/lib/cri-o-runc/sbin/runc
エラーコード
W0505 06:04:20.558166   18088 initconfiguration.go:120] Usage of CRI endpoints without URL scheme is deprecated and can cause kubelet errors in the future. Automatically prepending scheme "unix" to the "criSocket" with value "/usr/lib/cri-o-runc/sbin/runc". Please update your configuration!
[init] Using Kubernetes version: v1.27.1
[preflight] Running pre-flight checks
[preflight] The system verification failed. Printing the output from the verification:
KERNEL_VERSION: 5.15.102-1-pve
OS: Linux
CGROUPS_CPU: enabled
CGROUPS_CPUSET: enabled
CGROUPS_DEVICES: enabled
CGROUPS_FREEZER: enabled
CGROUPS_MEMORY: enabled
CGROUPS_PIDS: enabled
CGROUPS_HUGETLB: enabled
CGROUPS_IO: enabled
error execution phase preflight: [preflight] Some fatal errors occurred:
        [ERROR CRI]: container runtime is not running: output: E0505 06:04:21.154004   18121 remote_runtime.go:616] "Status from runtime service failed" err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial unix /usr/lib/cri-o-runc/sbin/runc: connect: connection refused\""
time="2023-05-05T06:04:21Z" level=fatal msg="getting status of runtime: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial unix /usr/lib/cri-o-runc/sbin/runc: connect: connection refused\""
, error: exit status 1
        [ERROR SystemVerification]: failed to parse kernel config: unable to load kernel module: "configs", output: "modprobe: FATAL: Module configs not found in directory /lib/modules/5.15.102-1-pve\n", err: exit status 1
        [ERROR FileContent--proc-sys-net-ipv4-ip_forward]: /proc/sys/net/ipv4/ip_forward contents are not set to 1
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher

proxmoxカーネルの検証エラーを無視する

https://zenn.dev/ddpn08/articles/d85176619f3fdd#エラー対応
を参考にした。
proxmoxのコンテナはカーネル名が/lib/modules/5.15.102-1-pveとなっており、kubeadmの未知のカーネルなのでエラーが出る。
これは--ignore-preflight-errors=SystemVerificationで無視する

kubeadm init --control-plane-endpoint 192.168.30.10 --cri-socket /usr/lib/cri-o-runc/sbin/runc --ignore-preflight-errors=SystemVerification
エラーコード
W0505 06:10:31.198314   18699 initconfiguration.go:120] Usage of CRI endpoints without URL scheme is deprecated and can cause kubelet errors in the future. Automatically prepending scheme "unix" to the "criSocket" with value "/usr/lib/cri-o-runc/sbin/runc". Please update your configuration!
[init] Using Kubernetes version: v1.27.1
[preflight] Running pre-flight checks
[preflight] The system verification failed. Printing the output from the verification:
KERNEL_VERSION: 5.15.102-1-pve
OS: Linux
CGROUPS_CPU: enabled
CGROUPS_CPUSET: enabled
CGROUPS_DEVICES: enabled
CGROUPS_FREEZER: enabled
CGROUPS_MEMORY: enabled
CGROUPS_PIDS: enabled
CGROUPS_HUGETLB: enabled
CGROUPS_IO: enabled
        [WARNING SystemVerification]: failed to parse kernel config: unable to load kernel module: "configs", output: "modprobe: FATAL: Module configs not found in directory /lib/modules/5.15.102-1-pve\n", err: exit status 1
error execution phase preflight: [preflight] Some fatal errors occurred:
        [ERROR CRI]: container runtime is not running: output: E0505 06:10:31.826183   18716 remote_runtime.go:616] "Status from runtime service failed" err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial unix /usr/lib/cri-o-runc/sbin/runc: connect: connection refused\""
time="2023-05-05T06:10:31Z" level=fatal msg="getting status of runtime: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial unix /usr/lib/cri-o-runc/sbin/runc: connect: connection refused\""
, error: exit status 1
        [ERROR FileContent--proc-sys-net-ipv4-ip_forward]: /proc/sys/net/ipv4/ip_forward contents are not set to 1
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher

ipv4 port forwardを設定する

https://access.redhat.com/documentation/ja-jp/red_hat_enterprise_linux/6/html/load_balancer_administration/s1-lvs-forwarding-vsa
を参考にした。
/sbin/sysctl net.ipv4.ip_forwardを設定する

ほなふくほなふく

調べてみると、CRI-Oが立ち上がっていないようだった。
そもそもコンテナの中でコンテナ技術であるCRI-Oを動かすのがだいぶ難しそうだった。
別のコンテナランタイムを使うことにする

ほなふくほなふく
wget https://github.com/containerd/containerd/releases/download/v1.6.20/containerd-1.6.20-linux-amd64.tar.gz
tar xvf containerd*.tar.gz
mv ./bin/* /usr/local/bin/ 
wget https://raw.githubusercontent.com/containerd/containerd/main/containerd.service
mkdir -p /usr/local/lib/systemd/system/
mv containerd.service /usr/local/lib/systemd/system/containerd.service
systemctl daemon-reload
systemctl enable --now containerd
ほなふくほなふく

runcのインストール

wget https://github.com/opencontainers/runc/releases/download/v1.1.7/runc.amd64
install -m 755 runc.amd64 /usr/local/sbin/runc
ほなふくほなふく

kubeadm init --control-plane-endpoint 192.168.30.10 --ignore-preflight-errors=SystemVerification

ほなふくほなふく
ln -s /dev/console /dev/kmsg
echo 'L /dev/kmsg - - - - /dev/console' > /etc/tmpfiles.d/kmsg.conf
ほなふくほなふく

vim /etc/pve/lxc/100.conf

lxc.apparmor.profile: unconfined
lxc.cap.drop:
lxc.cgroup.devices.allow: a
lxc.mount.auto: proc:rw sys:rw