Oracle Cloud上にTerraformで管理されたk8sクラスタを作る（WIP）

❯ oci iam availability-domain list   
ERROR: The config file at ~/.oci/config is invalid:

+Config Errors---+----------------------------------------------------------------------------+
| Key  | Error   | Hint                                                                       |
+------+---------+----------------------------------------------------------------------------+
| user | missing | log into the console and go to the user's settings page to find their OCID |
+------+---------+----------------------------------------------------------------------------+

❯ oci iam availability-domain list --auth security_token 
{
  "data": [
    {
      "compartment-id": "ocid1.tenancy.oc1..hogehoge,
      "id": "ocid1.availabilitydomain.oc1..hogehoge",
      "name": "HOGE:AP-OSAKA-1-AD-1"
    }
  ]
}

❯ tf plan
oci_core_vcn.test_vcn: Refreshing state... [id=ocid1.vcn.oc1.ap-osaka-1.amaaaaaalijvdeaa5qvriedazomylbc6u7to67iyn35y4ocov6bccduo2ooa]
╷
│ Error: 401-NotAuthenticated 
│ Provider version: 4.29.0, released on 2021-06-01.  
│ Service:  
│ Error Message: The required information to complete authentication was not provided or was incorrect. 

✗ oci session refresh                  
Attempting to refresh token from https://auth.ap-osaka-1.oraclecloud.com/v1/authentication/refresh
Your session is no longer valid and cannot be refreshed. Please use 'oci session authenticate' to create a new session.

oci session authenticate --profile-name DEFAULT --region ap-osaka-1

[opc@terraform-test ~]$ sudo cat > /etc/modules-load.d/containerd.conf <<EOF
> overlay
> br_netfilter
> EOF
-bash: /etc/modules-load.d/containerd.conf: Permission denied

[opc@terraform-test ~]$ sudo tee /etc/modules-load.d/containerd.conf <<EOF　>/dev/null
> overlay
> br_netfilter
> EOF

sudo kubeadm reset

sudo systemctl stop kubelet
sudo systemctl stop containerd
sudo rm -rf /var/lib/cni/
sudo rm -rf /var/lib/kubelet/*
sudo rm -rf /etc/cni/
sudo ifconfig cni0 down
sudo ifconfig flannel.1 down
sudo systemctl start kubelet
sudo systemctl start containerd

sudo tee /etc/modules-load.d/containerd.conf <<EOF >/dev/null
overlay
br_netfilter
EOF

sudo modprobe overlay
sudo modprobe br_netfilter
sudo sysctl --system

sudo yum install -y yum-utils device-mapper-persistent-data lvm2
sudo yum-config-manager \
    --add-repo \
    https://download.docker.com/linux/centos/docker-ce.repo
sudo yum update -y && sudo yum install -y containerd.io
sudo mkdir -p /etc/containerd
sudo containerd config default | sudo tee /etc/containerd/config.toml >/dev/null
sudo systemctl restart containerd

sudo tee /etc/yum.repos.d/kubernetes.repo <<EOF >/dev/null
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-aarch64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF

sudo setenforce 0
sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

sudo yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes

sudo systemctl enable --now kubelet

sudo systemctl daemon-reload
sudo systemctl restart kubelet

[opc@terraform-test ~]$ sudo kubeadm init --pod-network-cidr=10.244.0.0/16
[init] Using Kubernetes version: v1.21.1
[preflight] Running pre-flight checks
        [WARNING Firewalld]: firewalld is active, please ensure ports [6443 10250] are open or your cluster may not function correctly
        [WARNING Hostname]: hostname "terraform-test" could not be reached
        [WARNING Hostname]: hostname "terraform-test": lookup terraform-test on xxx.xxx.xxx.xxx:53: no such host
error execution phase preflight: [preflight] Some fatal errors occurred:
        [ERROR NumCPU]: the number of available CPUs 1 is less than the required 2
        [ERROR Swap]: running with swap on is not supported. Please disable swap
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher
[opc@terraform-test ~]$ 

[ERROR NumCPU]: the number of available CPUs 1 is less than the required 2
[ERROR Swap]: running with swap on is not supported. Please disable swap

[ERROR FileContent--proc-sys-net-ipv4-ip_forward]: /proc/sys/net/ipv4/ip_forward contents are not set to 1
[ERROR Swap]: running with swap on is not supported. Please disable swap

sudo tee /etc/sysctl.d/99-kubernetes-cri.conf <<EOF >/dev/null
net.bridge.bridge-nf-call-iptables  = 1
net.ipv4.ip_forward                 = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF

sudo sysctl --system

sudo swapoff -a
sudo sed -i 's/^\/.swapfile/#\/.swapfile/' /etc/fstab

[opc@k8s-cluster-arm-master-node ~]$ sudo kubeadm init --pod-network-cidr=10.244.0.0/16

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join xxx.xxx.xxx.xxx:xxxx --token xxxx \
        --discovery-token-ca-cert-hash sha256:xxxx 
[opc@k8s-cluster-arm-master-node ~]$ kubectl
kubectl controls the Kubernetes cluster manager.

 Find more information at: https://kubernetes.io/docs/reference/kubectl/overview/

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

[opc@k8s-cluster-arm-master-node ~]$ kubectl cluster-info
Kubernetes control plane is running at https://xxx.xxx.xxx.xxx:xxxx
CoreDNS is running at https://xxx.xxx.xxx.xxx:xxxx/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

[opc@k8s-cluster-arm-master-node ~]$ kubectl describe nodes

...

Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Sun, 06 Jun 2021 14:47:00 +0000   Sun, 06 Jun 2021 14:36:35 +0000   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Sun, 06 Jun 2021 14:47:00 +0000   Sun, 06 Jun 2021 14:36:35 +0000   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Sun, 06 Jun 2021 14:47:00 +0000   Sun, 06 Jun 2021 14:36:35 +0000   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            False   Sun, 06 Jun 2021 14:47:00 +0000   Sun, 06 Jun 2021 14:36:35 +0000   KubeletNotReady              container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized

sudo firewall-cmd --add-port=80/tcp --permanent
sudo firewall-cmd --add-port=443/tcp --permanent
sudo firewall-cmd --add-port=6443/tcp --permanent
sudo firewall-cmd --add-port=2379/tcp --permanent
sudo firewall-cmd --add-port=2380/tcp --permanent
sudo firewall-cmd --add-port=10250/tcp --permanent
sudo firewall-cmd --add-port=10251/tcp --permanent
sudo firewall-cmd --add-port=10252/tcp --permanent
sudo firewall-cmd --add-port=30000-32767/tcp --permanent
sudo firewall-cmd --reload