👻
Proxmoxを使ってお家Kubernetesクラスター構築
ある程度ちゃんとしたKubernetesのクラスター環境を構築しようとすると、やっぱり物理サーバが最低でも2台欲しいところ。。。
選択肢としては、次の３パターンのいづれかを採用したい。
- 2台のラズベリーパイ用意し、Kubernetesクラスターを構築する
- 1台の物理マシンにProxmoxを構築し、その上に2台のUbuntuマシンを立てKubernetesクラスターを構築する
- パターン2のOpenStack版
今回はパターン2を採用しました。
採用した背景として、まずお金をかけたくありませんでした。
そのため、新たにラズパイを2台購入するのはコストがかかるので選択肢から外れました。
次に、構築のし易さを考えたときに、OpenStackの構築は手間がかかるなと想像し、選択肢から外しました。
よって、お金をかけずに、簡単に構築できそうなパターン2を採用しました。
Proxmox
Proxmoxのインストール
以前勉強用に購入したミニサーバがあるのでそちらのサーバにインストールしました。
インストール手順については、Proxmox VEのセットアップ手順（インストール編）
がおすすめです。
インストール後の初期設定
IPアドレス固定
個人的にDHCPでコロコロIPアドレス変わるの好きじゃないので、固定しました。
/etc/netplan/00-installer-config.yaml
# This is the network config written by 'subiquity'
network:
ethernets:
ens18:
addresses: [192.168.0.XX/24]
routes:
- to: default
via: 192.168.0.1
nameservers:
addresses: [192.168.0.1 , 8.8.8.8]
search: []
optional: true
00-installer-config.yamlファイルを修正後、netplan applyを実行
netplan apply
Kubernetesクラスター構築
Proxmox上で2台のUbuntuインスタンスを作成します。
1つはマスターノード、もう1つはワーカーノードとして動作させます。
2台とも同じスペックです
| パラメータ | 値 |
|---|---|
| OS | Ubuntu22.04 |
| Disk Size | 32GB |
| Processors | 1 sockets, 2 cores |
| Memory | 4GB |
kubeadmのインストール
このセクションでは、全てのマシン(2台のUbuntuインスタンス両方)で実施します。
基本的に公式ドキュメントに沿って構築します。
kubeadmをインストールする前にやること
swapのoff
sudo su -
swapoff -a
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point> <type> <options> <dump> <pass>
# / was on /dev/ubuntu-vg/ubuntu-lv during curtin installation
/dev/disk/by-id/dm-uuid-LVM-aq2ElNNLksret98GNCi7z2unyz45SQG7UPhnoYybdPWsfecRlnAsqttOfjXyZW1l / ext4 defaults 0 1
# /boot was on /dev/sda2 during curtin installation
/dev/disk/by-uuid/79e7999d-c582-4161-a102-1e18c2610220 /boot ext4 defaults 0 1
#/swap.img none swap sw 0 0 #この行をコメントアウト
IPv4フォワーディングを有効化し、iptablesからブリッジされたトラフィックを見えるようにする
公式ドキュメントにコンテナランタイムのインストール要件に記載の設定を実施します。
cat <<EOF > /etc/modules-load.d/crio.conf
overlay
br_netfilter
EOF
cat <<EOF > /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
設定反映
modprobe overlay
modprobe br_netfilter
sysctl --system
iptables が nftables バックエンドを使用しないようにする(おまじない)
# レガシーバイナリをインストール
sudo apt-get install -y iptables arptables ebtables
# レガシーバージョンに切り替える
sudo update-alternatives --set iptables /usr/sbin/iptables-legacy
sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy
sudo update-alternatives --set arptables /usr/sbin/arptables-legacy
sudo update-alternatives --set ebtables /usr/sbin/ebtables-legacy
kubelet kubeadm kubectl CRI-Oのインストール
公式ドキュメントをそのまま実行します。
apt-get update
apt-get install -y software-properties-common curl
KUBERNETES_VERSION=v1.30
PROJECT_PATH=prerelease:/main
curl -fsSL https://pkgs.k8s.io/core:/stable:/$KUBERNETES_VERSION/deb/Release.key | gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/$KUBERNETES_VERSION/deb/ /" | tee /etc/apt/sources.list.d/kubernetes.list
curl -fsSL https://pkgs.k8s.io/addons:/cri-o:/$PROJECT_PATH/deb/Release.key | gpg --dearmor -o /etc/apt/keyrings/cri-o-apt-keyring.gpg
echo "deb [signed-by=/etc/apt/keyrings/cri-o-apt-keyring.gpg] https://pkgs.k8s.io/addons:/cri-o:/$PROJECT_PATH/deb/ /" | tee /etc/apt/sources.list.d/cri-o.list
apt-get update
apt-get install -y cri-o kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl
systemctl start crio.service
systemctl enable crio.service
systemctl status crio.service
コントロールプレーンマシンでの作業
sudo kubeadm init --pod-network-cidr=10.244.0.0/16
実行ログ
root@master:~# sudo kubeadm init --pod-network-cidr=10.244.0.0/16
I0616 11:15:40.622959 5874 version.go:256] remote version is much newer: v1.30.2; falling back to: stable-1.28
[init] Using Kubernetes version: v1.28.11
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local master] and IPs [10.96.0.1 192.168.0.18]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [localhost master] and IPs [192.168.0.18 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [localhost master] and IPs [192.168.0.18 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 5.501469 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node master as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node master as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule]
[bootstrap-token] Using token: tkkv1b.u58huw4hsv036ksu
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 192.168.0.18:6443 --token tkkv1b.u****6ksu \
--discovery-token-ca-cert-hash sha256:f40fb00ec871284089aaa93f78cd10a8e2baff47b3d626519ffe987c7b393879
root@master:~#
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
root@master:~# export KUBECONFIG=/etc/kubernetes/admin.conf
ワーカーノードでの作業
kubeadm join 192.168.0.4:6443 --token 07g9uo.0mowtsa8irzo1yuu \
--discovery-token-ca-cert-hash sha256:8941236ce5f1b8cbfce7885d4f2fe24820d2b3e996278729e793c61e33af5aa1
実行ログ
root@worker:~# kubeadm join 192.168.0.18:6443 --token tkkv1b.u****6ksu --discovery-token-ca-cert-hash sha256:f40fb00ec871284089aaa93f78cd10a8e2baff47b3d626519ffe987c7b393879
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
root@worker:~#
ワーカーノードが追加されたことを確認
root@master:~# kubectl get node
NAME STATUS ROLES AGE VERSION
master Ready control-plane 7m49s v1.28.11
worker Ready <none> 2m14s v1.28.11
root@master:~#
workerノードのROLESが<none>をworkerとかに設定する。
root@master:~# kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
master Ready control-plane 19m v1.30.2 192.168.0.4 <none> Ubuntu 22.04.4 LTS 5.15.0-113-generic cri-o://1.31.0
worker Ready <none> 16m v1.30.2 192.168.0.5 <none> Ubuntu 22.04.4 LTS 5.15.0-113-generic cri-o://1.31.0
root@master:~# kubectl label node worker node-role.kubernetes.io/worker=worker
node/worker labeled
root@master:~# kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
master Ready control-plane 21m v1.30.2 192.168.0.4 <none> Ubuntu 22.04.4 LTS 5.15.0-113-generic cri-o://1.31.0
worker Ready worker 19m v1.30.2 192.168.0.5 <none> Ubuntu 22.04.4 LTS 5.15.0-113-generic cri-o://1.31.0
root@master:~#
Calicoをデプロイ
Tigera Calicoオペレータとカスタムリソース定義をインストールします
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.28.0/manifests/tigera-operator.yaml
必要なカスタムリソースを作成して Calico をインストールします
このマニフェストをwgetで落とします。
wget https://raw.githubusercontent.com/projectcalico/calico/v3.28.0/manifests/custom-resources.yaml
custom-resources.yamlのCIDRを編集します。
custom-resources.yaml
# This section includes base Calico installation configuration.
# For more information, see: https://docs.tigera.io/calico/latest/reference/installation/api#operator.tigera.io/v1.Installation
apiVersion: operator.tigera.io/v1
kind: Installation
metadata:
name: default
spec:
# Configures Calico networking.
calicoNetwork:
ipPools:
- name: default-ipv4-ippool
blockSize: 26
cidr: 10.244.0.0/16 #このアドレス範囲を修正
encapsulation: VXLANCrossSubnet
natOutgoing: Enabled
nodeSelector: all()
---
# This section configures the Calico API server.
# For more information, see: https://docs.tigera.io/calico/latest/reference/installation/api#operator.tigera.io/v1.APIServer
apiVersion: operator.tigera.io/v1
kind: APIServer
metadata:
name: default
spec: {}
Calicoのリソースを下記コマンドで作成します。
kubectl apply -f custom-resources.yaml
Calicoのリソースが動いていることを確認。新たなネームスペースcalico-systemにPodがデプロイされる。
体感5~10分くらい待ちます。
root@master:~# kubectl get pod -n calico-system -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
calico-kube-controllers-746bdc5cfb-gxvl6 1/1 Running 0 3m23s 10.85.0.3 worker <none> <none>
calico-node-kv225 1/1 Running 0 3m24s 192.168.0.19 worker <none> <none>
calico-node-rc7sc 1/1 Running 0 3m24s 192.168.0.18 master <none> <none>
calico-typha-79d987c9c8-2b9nz 1/1 Running 0 3m24s 192.168.0.19 worker <none> <none>
csi-node-driver-nssqf 2/2 Running 0 3m23s 10.85.0.4 master <none> <none>
csi-node-driver-tbsjn 2/2 Running 0 3m23s 10.85.0.2 worker <none> <none>
kube-systemのPodの動作状況も確認
root@master:~# kubectl get pods -n kube-system -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
coredns-7db6d8ff4d-jtmmw 1/1 Running 0 14m 10.85.0.2 master <none> <none>
coredns-7db6d8ff4d-vwrlp 1/1 Running 0 14m 10.85.0.3 master <none> <none>
etcd-master 1/1 Running 0 15m 192.168.0.4 master <none> <none>
kube-apiserver-master 1/1 Running 0 15m 192.168.0.4 master <none> <none>
kube-controller-manager-master 1/1 Running 0 15m 192.168.0.4 master <none> <none>
kube-proxy-8trfr 1/1 Running 0 12m 192.168.0.5 worker <none> <none>
kube-proxy-v85cr 1/1 Running 0 14m 192.168.0.4 master <none> <none>
kube-scheduler-master 1/1 Running 0 15m 192.168.0.4 master <none> <none>
root@master:~#
環境構築後のPod間ネットワークの疎通確認
NginxのPodを冗長構成のために3つ作成します。
kubectl create deployment nginx --image=nginx --replicas=3
PodのIPアドレスを確認すると、custom-resources.yamlで定義したIPアドレスの範囲が割当たっています。
root@master:~# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-7854ff8877-6dtpx 1/1 Running 0 14s 10.244.171.68 worker <none> <none>
nginx-7854ff8877-kwjxs 1/1 Running 0 14s 10.244.171.67 worker <none> <none>
nginx-7854ff8877-xr2qd 1/1 Running 0 14s 10.244.171.66 worker <none> <none>
root@master:~#
Pod間で通信が可能か確認するため、とあるPodからアクセスしてみる
root@master:~# kubectl exec nginx-7854ff8877-6dtpx -- curl 10.244.171.67
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 615 100 615 0 0 308k 0 --:--:-- --:--:-- --:--:-- 600k
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
root@master:~# kubectl exec nginx-7854ff8877-6dtpx -- curl 10.244.171.66
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 615 100 615 0 0 621k 0 --:--:-- --:--:-- --:--:-- 600k
root@master:~#
Discussion