Closed39

Oracle Cloud上にTerraformで管理されたk8sクラスタを作る(WIP)

Futa HirakobaFuta Hirakoba

いろいろメモ

Futa HirakobaFuta Hirakoba

oci-cliにおいてsecurity_tokenで認証している場合、--authで設定しないといけない。

設定なし
❯ oci iam availability-domain list   
ERROR: The config file at ~/.oci/config is invalid:

+Config Errors---+----------------------------------------------------------------------------+
| Key  | Error   | Hint                                                                       |
+------+---------+----------------------------------------------------------------------------+
| user | missing | log into the console and go to the user's settings page to find their OCID |
+------+---------+----------------------------------------------------------------------------+
--auth設定
❯ oci iam availability-domain list --auth security_token 
{
  "data": [
    {
      "compartment-id": "ocid1.tenancy.oc1..hogehoge,
      "id": "ocid1.availabilitydomain.oc1..hogehoge",
      "name": "HOGE:AP-OSAKA-1-AD-1"
    }
  ]
}
Futa HirakobaFuta Hirakoba

時間経つとセッション切れる

❯ tf plan
oci_core_vcn.test_vcn: Refreshing state... [id=ocid1.vcn.oc1.ap-osaka-1.amaaaaaalijvdeaa5qvriedazomylbc6u7to67iyn35y4ocov6bccduo2ooa]
╷
│ Error: 401-NotAuthenticated 
│ Provider version: 4.29.0, released on 2021-06-01.  
│ Service:  
│ Error Message: The required information to complete authentication was not provided or was incorrect. 

リフレッシュ

✗ oci session refresh                  
Attempting to refresh token from https://auth.ap-osaka-1.oraclecloud.com/v1/authentication/refresh
Your session is no longer valid and cannot be refreshed. Please use 'oci session authenticate' to create a new session.

リフレッシュも時間経ちすぎるとダメだとか

再認証

oci session authenticate --profile-name DEFAULT --region ap-osaka-1
Futa HirakobaFuta Hirakoba

進捗

Futa HirakobaFuta Hirakoba

k8sインストール手順をとりあえず手動実行

Futa HirakobaFuta Hirakoba

必要な設定の追加

sudo tee /etc/modules-load.d/containerd.conf <<EOF >/dev/null
overlay
br_netfilter
EOF

sudo modprobe overlay
sudo modprobe br_netfilter
sudo sysctl --system
Futa HirakobaFuta Hirakoba

containerdのインストール

sudo yum install -y yum-utils device-mapper-persistent-data lvm2
sudo yum-config-manager \
    --add-repo \
    https://download.docker.com/linux/centos/docker-ce.repo
sudo yum update -y && sudo yum install -y containerd.io
sudo mkdir -p /etc/containerd
sudo containerd config default | sudo tee /etc/containerd/config.toml >/dev/null
sudo systemctl restart containerd
Futa HirakobaFuta Hirakoba

baseurlはarm向けのものにする

sudo tee /etc/yum.repos.d/kubernetes.repo <<EOF >/dev/null
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-aarch64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF

sudo setenforce 0
sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

sudo yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes

sudo systemctl enable --now kubelet
sudo systemctl daemon-reload
sudo systemctl restart kubelet
Futa HirakobaFuta Hirakoba

マスターノード初期化

[opc@terraform-test ~]$ sudo kubeadm init --pod-network-cidr=10.244.0.0/16
[init] Using Kubernetes version: v1.21.1
[preflight] Running pre-flight checks
        [WARNING Firewalld]: firewalld is active, please ensure ports [6443 10250] are open or your cluster may not function correctly
        [WARNING Hostname]: hostname "terraform-test" could not be reached
        [WARNING Hostname]: hostname "terraform-test": lookup terraform-test on xxx.xxx.xxx.xxx:53: no such host
error execution phase preflight: [preflight] Some fatal errors occurred:
        [ERROR NumCPU]: the number of available CPUs 1 is less than the required 2
        [ERROR Swap]: running with swap on is not supported. Please disable swap
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher
[opc@terraform-test ~]$ 

失敗。

[ERROR NumCPU]: the number of available CPUs 1 is less than the required 2
[ERROR Swap]: running with swap on is not supported. Please disable swap

それぞれ調査する

Futa HirakobaFuta Hirakoba

CPUの数増やして再度sudo kubeadm init --pod-network-cidr=10.244.0.0/16に来た

[ERROR FileContent--proc-sys-net-ipv4-ip_forward]: /proc/sys/net/ipv4/ip_forward contents are not set to 1
[ERROR Swap]: running with swap on is not supported. Please disable swap
Futa HirakobaFuta Hirakoba

ipv4を1にする設定を追加

sudo tee /etc/sysctl.d/99-kubernetes-cri.conf <<EOF >/dev/null
net.bridge.bridge-nf-call-iptables  = 1
net.ipv4.ip_forward                 = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF

sudo sysctl --system
Futa HirakobaFuta Hirakoba

swapを無効化。
/etc/fstabを書き換えて永続的に無効化

sudo swapoff -a
sudo sed -i 's/^\/.swapfile/#\/.swapfile/' /etc/fstab
Futa HirakobaFuta Hirakoba

master-node建った

[opc@k8s-cluster-arm-master-node ~]$ sudo kubeadm init --pod-network-cidr=10.244.0.0/16

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join xxx.xxx.xxx.xxx:xxxx --token xxxx \
        --discovery-token-ca-cert-hash sha256:xxxx 
[opc@k8s-cluster-arm-master-node ~]$ kubectl
kubectl controls the Kubernetes cluster manager.

 Find more information at: https://kubernetes.io/docs/reference/kubectl/overview/
Futa HirakobaFuta Hirakoba

設定。

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

キタキタ

[opc@k8s-cluster-arm-master-node ~]$ kubectl cluster-info
Kubernetes control plane is running at https://xxx.xxx.xxx.xxx:xxxx
CoreDNS is running at https://xxx.xxx.xxx.xxx:xxxx/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
Futa HirakobaFuta Hirakoba

うおーーーnodeがNotReadyだった〜

[opc@k8s-cluster-arm-master-node ~]$ kubectl describe nodes

...

Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Sun, 06 Jun 2021 14:47:00 +0000   Sun, 06 Jun 2021 14:36:35 +0000   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Sun, 06 Jun 2021 14:47:00 +0000   Sun, 06 Jun 2021 14:36:35 +0000   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Sun, 06 Jun 2021 14:47:00 +0000   Sun, 06 Jun 2021 14:36:35 +0000   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            False   Sun, 06 Jun 2021 14:47:00 +0000   Sun, 06 Jun 2021 14:36:35 +0000   KubeletNotReady              container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
Futa HirakobaFuta Hirakoba

ポート解放する。Oracle-Linux-8.3-aarch64-2021.05.12-0 ではiptablesではなくfirewalldが使われている。sudo firewall-cmd --list-allで現在の設定を確認する

sudo firewall-cmd --add-port=80/tcp --permanent
sudo firewall-cmd --add-port=443/tcp --permanent
sudo firewall-cmd --add-port=6443/tcp --permanent
sudo firewall-cmd --add-port=2379/tcp --permanent
sudo firewall-cmd --add-port=2380/tcp --permanent
sudo firewall-cmd --add-port=10250/tcp --permanent
sudo firewall-cmd --add-port=10251/tcp --permanent
sudo firewall-cmd --add-port=10252/tcp --permanent
sudo firewall-cmd --add-port=30000-32767/tcp --permanent
sudo firewall-cmd --reload
Futa HirakobaFuta Hirakoba

ポートが開かない。バックグラウンドはnftablesだけど、なーぜか設定が適用されない

Futa HirakobaFuta Hirakoba

centos8でnftablesをiptablesに切り替えるのキツいためubuntuを使いたかったがoracle cloudのイメージにARMベースのubuntuがなかった...
しょうがないのでcentos7.9にした

このスクラップは2022/09/21にクローズされました