Zenn
Open13

Mac+Ubuntu 22.04 メモ・おうちk8sクラスタ計画

Kubernetes
kubectl
kubeadm
Ubuntu 22.04
TAK848TAK848

後から見返して，なにやったか忘れない程度にメモしておく

TAK848TAK848

ここからk8s

install docker

https://docs.docker.com/engine/install/ubuntu/

for pkg in docker.io docker-doc docker-compose podman-docker containerd runc; do sudo apt-get remove $pkg; done

# Add Docker's official GPG key:
sudo apt-get update
sudo apt-get install ca-certificates curl gnupg -y
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg

# Add the repository to Apt sources:
echo \
  "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update

sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin -y

sudo docker run hello-world

https://docs.docker.com/engine/install/linux-postinstall/

sudo usermod -aG docker $USER
TAK848TAK848

kubeadmなどインストール

https://kubernetes.io/ja/docs/setup/production-environment/tools/kubeadm/install-kubeadm/

# ポートがあいてるか確認
nc 127.0.0.1 6443

# install
sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl

curl -fsSL https://dl.k8s.io/apt/doc/apt-key.gpg | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-archive-keyring.gpg

echo "deb [signed-by=/etc/apt/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list

sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl

containerdの設定

やる

https://kubernetes.io/docs/setup/production-environment/container-runtimes/#install-and-configure-prerequisites

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

sudo modprobe overlay
sudo modprobe br_netfilter

# sysctl params required by setup, params persist across reboots
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF

# Apply sysctl params without reboot
sudo sysctl --system

special thanks

このままkubeadm init --v=5すると，

I0910 23:49:52.025096   17712 clusterinfo.go:84] creating the RBAC rules for exposing the cluster-info ConfigMap in the kube-public namespace
I0910 23:49:52.181544   17712 kubeletfinalize.go:90] [kubelet-finalize] Assuming that kubelet client certificate rotation is enabled: found "/var/lib/kubelet/pki/kubelet-client-current.pem"
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
I0910 23:49:52.182157   17712 kubeletfinalize.go:134] [kubelet-finalize] Restarting the kubelet to enable client certificate rotation
rpc error: code = Unknown desc = malformed header: missing HTTP content-type
unable to create RBAC clusterrolebinding
...

とエラー。
api serverが動かないらしい？？

以下で解決できた🙇‍♂️
https://www.notr.app/posts/2023/07/17/
https://github.com/containerd/containerd/issues/8139#issuecomment-1536805958

# containerdのconfigの初期化
containerd config default | sudo tee /etc/containerd/config.toml
# runcのcgroupドライバー
sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/g' /etc/containerd/config.toml

sudo systemctl restart containerd
TAK848TAK848

kubeadm起動

https://kubernetes.io/ja/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/

$ sudo kubeadm init --pod-network-cidr=10.144.0.0/14
[init] Using Kubernetes version: v1.28.1
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
...
...
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join a.b.c.d:6443 --token XXXXXXXXXXXXXXXX \
	--discovery-token-ca-cert-hash sha256:YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY
TAK848TAK848

設定

言われたとおりに

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

CNI install (Flannel)

https://github.com/flannel-io/flannel#deploying-flannel-manually

kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml

しばらく待つと

$ kubectl get nodes
NAME              STATUS     ROLES           AGE   VERSION
x   NotReady   control-plane   18m   v1.28.1
$ kubectl get nodes
NAME              STATUS   ROLES           AGE   VERSION
x   Ready    control-plane   18m   v1.28.1

やったね！と思いきや

$ kubectl get pods -A
NAMESPACE      NAME                                      READY   STATUS              RESTARTS      AGE
kube-flannel   kube-flannel-ds-x7cll                     0/1     CrashLoopBackOff    4 (35s ago)   2m24s
kube-system    coredns-5dd5756b68-5xvlm                  0/1     ContainerCreating   0             20m
kube-system    coredns-5dd5756b68-7584v                  0/1     ContainerCreating   0             20m

ログを見ると

...
I0910 15:15:35.016546       1 vxlan.go:141] VXLAN config: VNI=1 Port=0 GBP=false Learning=false DirectRouting=false
E0910 15:15:35.016758       1 main.go:335] Error registering network: failed to acquire lease: subnet "10.244.0.0/16" specified in the flannel net config doesn't contain "10.144.0.0/24" PodCIDR of the "tak-imac-ubuntu" node
I0910 15:15:35.016795       1 main.go:523] Stopping shutdownHandler...

あれ，CIDR変えてるのに反映されてない...
flannel.ymlに直書きされてるのかな？->されてた

--pod-network-cidr=10.144.0.0/14のように，pod-network-cidrを変更して起動したため，flannelのyamlをそのまま適用はできなかった
flannel.ymlを

  net-conf.json: |
    {
-      "Network": "10.244.0.0/16",
+      "Network": "10.144.0.0/14",
      "Backend": {
        "Type": "vxlan"
      }
    }

のように書き換えて解決！

$ kubectl apply -f flannel.yaml
$ kubectl get pods -A
NAMESPACE      NAME                                      READY   STATUS    RESTARTS   AGE
kube-flannel   kube-flannel-xx-xxxxx                     1/1     Running   0          4m37s
kube-system    coredns-xxxxxxxxxx-xxxxx                  1/1     Running   0          30m
kube-system    coredns-xxxxxxxxxx-xxxxx                  1/1     Running   0          30m

ラズパイでエラー

ラズパイ上でのflannelが，Errorで死んでいる

...
I0910 23:06:04.791187       1 main.go:543] Found network config - Backend type: vxlan
I0910 23:06:04.791335       1 match.go:206] Determining IP address of default interface
I0910 23:06:04.793606       1 match.go:259] Using interface with name wlan0 and address 10.0.5.1
I0910 23:06:04.793761       1 match.go:281] Defaulting external address to interface address (10.0.5.1)
I0910 23:06:04.794045       1 vxlan.go:141] VXLAN config: VNI=1 Port=0 GBP=false Learning=false DirectRouting=false
E0910 23:06:04.803935       1 main.go:335] Error registering network: operation not supported
I0910 23:06:04.804125       1 main.go:523] Stopping shutdownHandler...

https://qiita.com/showchan33/items/5250f518eb03858a0c25

sudo apt update -y && \
  sudo apt install -y linux-modules-extra-raspi

再度適用

TAK848TAK848

master node(control plane)にnginxのDeploymentをデプロイ・アクセス

設定

master nodeにもPod配置させてくれ（おうちでそんなリソース贅沢じゃないので…

https://qiita.com/KTwawawa/items/d51a056b724ad73d1e7a

$ kubectl describe node | grep Taints
Taints:             node-role.kubernetes.io/control-plane:NoSchedule
$ kubectl get node
NAME              STATUS   ROLES           AGE   VERSION
xxxxxxx   Ready    control-plane   35m   v1.28.1
$ kubectl taint nodes xxxxxxx node-role.kubernetes.io/control-plane:NoSchedule-
node/tak-imac-ubuntu untainted
$ kubectl describe node | grep Taints
Taints:             <none>

作成

$ kubectl create deployment nginx --image=nginx
deployment.apps/nginx created
$ kubectl create service nodeport nginx --tcp=80:80
service/nginx created
$ kubectl get service
NAME         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
kubernetes   ClusterIP   10.96.0.1      <none>        443/TCP        38m
nginx        NodePort    10.109.18.63   <none>        80:31922/TCP   9s
$ curl localhost:31922
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
$ kubectl get pods -owide
NAME                     READY   STATUS    RESTARTS   AGE   IP           NODE              NOMINATED NODE   READINESS GATES
nginx-7854ff8877-fkmz4   1/1     Running   0          76s   10.144.0.4   xxxxxxx   <none>           <none>

掃除

$ kubectl get deployment
NAME    READY   UP-TO-DATE   AVAILABLE   AGE
nginx   1/1     1            1           3m33s
$ kubectl delete deployment nginx
deployment.apps "nginx" deleted
$ kubectl get service
NAME         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
kubernetes   ClusterIP   10.96.0.1      <none>        443/TCP        41m
nginx        NodePort    10.109.18.63   <none>        80:31922/TCP   3m10s
$ kubectl delete service nginx
service "nginx" deleted
TAK848TAK848

CPを3台にする

とりあえず各ノードで破壊

sudo kubeadm reset

haproxyでapiserverをバランシングする。

sudo apt update && sudo apt install haproxy -y
sudo cp /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg.bak
sudo vim /etc/haproxy/haproxy.cfg

global
    log /dev/log local0
    log /dev/log local1 notice
    daemon
defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 1
    timeout http-request    10s
    timeout queue           20s
    timeout connect         5s
    timeout client          60m
    timeout server          60m
    timeout http-keep-alive 10s
    timeout check           10s

frontend kubernetes-frontend
    bind *:6443
    mode tcp
    default_backend kubernetes-backend

backend kubernetes-backend
    option httpchk GET /healthz
    http-check expect status 200
    mode tcp
    balance roundrobin
    option tcp-check
    tcp-check connect port 6443
    server master1 master1:6443 check fall 3 rise 2
    server master2 master2:6443 check fall 3 rise 2
    server master3 master3:6443 check fall 3 rise 2

sudo systemctl restart haproxy
TAK848TAK848
DNS設定いじった
sudo vim /etc/systemd/resolved.conf

追記

[Resolve]
DNS=10.2.1.1
FallbackDNS=

sudo systemctl restart systemd-resolved
TAK848TAK848

マルチCPでやりなおす

sudo kubeadm init --pod-network-cidr=10.244.0.0/16  --control-plane-endpoint "k8s-proxy.tak848.net:6443" --upload-certs

他では，kubeadm joinしていく

resolv.confの場所変更

systemd-resolvedを無効化してカスタムのDNSをホストしてるノードでは，以下を実行

$ sudo vim /var/lib/kubelet/kubeadm-flags.env

- KUBELET_KUBEADM_ARGS="--container-runtime-endpoint=unix:///var/run/containerd/containerd.sock --pod-infra-container-image=registry.k8s.io/pause:3.9"
+ KUBELET_KUBEADM_ARGS="--resolv-conf=/etc/resolv.conf --container-runtime-endpoint=unix:///var/run/containerd/containerd.sock --pod-infra-container-image=registry.k8s.io/pause:3.9"

sudo systemctl restart kubelet

flannelをhelmで入れる

# Needs manual creation of namespace to avoid helm error
kubectl create ns kube-flannel
kubectl label --overwrite ns kube-flannel pod-security.kubernetes.io/enforce=privileged

helm repo add flannel https://flannel-io.github.io/flannel/
helm install flannel --set podCidr="10.244.0.0/16" --namespace kube-flannel flannel/flannel
TAK848TAK848

色々入れていく

nginx ingress controller

node指定

values.yaml
controller:
  nodeSelector:
    kubernetes.io/hostname: xxxxxxxx

$ helm upgrade --install ingress-nginx ingress-nginx \
  --repo https://kubernetes.github.io/ingress-nginx \
  --namespace ingress-nginx --create-namespace -f values.yaml 
NAME: ingress-nginx
LAST DEPLOYED: 
NAMESPACE: ingress-nginx
STATUS: deployed
REVISION: 4
TEST SUITE: None
NOTES:
The ingress-nginx controller has been installed.
It may take a few minutes for the LoadBalancer IP to be available.
You can watch the status by running 'kubectl --namespace ingress-nginx get services -o wide -w ingress-nginx-controller'

An example Ingress that makes use of the controller:
  apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    name: example
    namespace: foo
  spec:
    ingressClassName: nginx
    rules:
      - host: www.example.com
        http:
          paths:
            - pathType: Prefix
              backend:
                service:
                  name: exampleService
                  port:
                    number: 80
              path: /
    # This section is only required if TLS is to be enabled for the Ingress
    tls:
      - hosts:
        - www.example.com
        secretName: example-tls

If TLS is enabled for the Ingress, a Secret containing the certificate and key must also be provided:

  apiVersion: v1
  kind: Secret
  metadata:
    name: example-tls
    namespace: foo
  data:
    tls.crt: <base64 encoded cert>
    tls.key: <base64 encoded key>
  type: kubernetes.io/tls

https://kubernetes.github.io/ingress-nginx/deploy/#quick-start

cert-manager

https://cert-manager.io/docs/installation/helm/#steps

helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install \
  cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --version v1.13.0 \
  --set installCRDs=true

MetalLB

https://metallb.universe.tf/installation/#installation-with-helm

helm repo add metallb https://metallb.github.io/metallb
helm install metallb metallb/metallb