👌
kubesprayを使ってAWSEC2群にkubernetesクラスタをインストールする
kubesprayとは
- kubernetesスタックのインストーラー
- aws、Openstach、ベアメタル、主要Linuxディストロなど、さまざまなOSにKubernetesをインストールすることができるansible playbook
- https://github.com/kubernetes-sigs/kubespray
実行完了
- Ubuntu22.04(wsl2)
手順
セットアップ
リポジトリクローン
g clone https://github.com/kubernetes-sigs/kubespray
cd kubespray
kubesplay実行環境セットアップ
- 最低限、実行マシンにpythonが入っていれば良い。
python3 -m venv vent
source venv/bin/activate
pip install -r requirements.txt
検証用環境構築
- AWSが手っ取り早そう
terraformでの構成セットが入っているディレクトリへ移動
cd ./contrib/terraform/aws/
Versionを指定。少なくとも0.12.23では動作した。
最新の1.x.xでは動作しないかもしれません。
tfenv use 0.12.23
クレデンシャルファイルをコピペ
cp credentials.tfvars.example credentials.tfvars
パラメータをセットする。
AWS_SSH_KEY_NAMEはあらかじめ、コンソールなどでキーペアを作って、それを指定した
#AWS Access Key
AWS_ACCESS_KEY_ID = "xxxxxxxx"
#AWS Secret Key
AWS_SECRET_ACCESS_KEY = "xxxxxxxx"
#EC2 SSH Key Name
AWS_SSH_KEY_NAME = "k8s"
#AWS Region
AWS_DEFAULT_REGION = "ap-northeast-1"
credentials.tfvarsを引数にterraform apply実行
terraform apply -var-file=credentials.tfvars
terraform applyが完了すると、inventory/hostsに、各EC2のFQDNがセットされる。
のちの、Ansible Playbook実行時に指定することになります。
[all]
ip-10-250-197-20.ap-northeast-1.compute.internal ansible_host=10.250.197.20
ip-10-250-221-144.ap-northeast-1.compute.internal ansible_host=10.250.221.144
ip-10-250-207-144.ap-northeast-1.compute.internal ansible_host=10.250.207.144
ip-10-250-198-1.ap-northeast-1.compute.internal ansible_host=10.250.198.1
ip-10-250-210-49.ap-northeast-1.compute.internal ansible_host=10.250.210.49
ip-10-250-199-180.ap-northeast-1.compute.internal ansible_host=10.250.199.180
ip-10-250-218-202.ap-northeast-1.compute.internal ansible_host=10.250.218.202
bastion ansible_host=3.113.30.102
[bastion]
bastion ansible_host=3.113.30.102
[kube_control_plane]
ip-10-250-197-20.ap-northeast-1.compute.internal
ip-10-250-221-144.ap-northeast-1.compute.internal
ip-10-250-207-144.ap-northeast-1.compute.internal
[kube_node]
ip-10-250-198-1.ap-northeast-1.compute.internal
ip-10-250-210-49.ap-northeast-1.compute.internal
ip-10-250-199-180.ap-northeast-1.compute.internal
ip-10-250-218-202.ap-northeast-1.compute.internal
[etcd]
ip-10-250-197-20.ap-northeast-1.compute.internal
ip-10-250-221-144.ap-northeast-1.compute.internal
ip-10-250-207-144.ap-northeast-1.compute.internal
[calico_rr]
[k8s_cluster:children]
kube_node
kube_control_plane
calico_rr
[k8s_cluster:vars]
apiserver_loadbalancer_domain_name="kubernetes-nlb-devtest-035904b132ec747b.elb.ap-northeast-1.amazonaws.com"
kubernetesインストール
ssh-keyをssh-agentにセット
eval (ssh-agent -c)
ssh-add -D
ssh-add ~/.ssh/k8s.pem
ssh-add -L
構築コマンド
ansible-playbook -i ./inventory/hosts ./cluster.yml -e ansible_user=admin -b --become-user=root --flush-cache
完了後の出力
PLAY RECAP *****************************************************************************************************************************************************************************************************************************************************************************
bastion : ok=7 changed=1 unreachable=0 failed=0 skipped=13 rescued=0 ignored=0
ip-10-250-197-20.ap-northeast-1.compute.internal : ok=697 changed=147 unreachable=0 failed=0 skipped=1145 rescued=0 ignored=8
ip-10-250-198-1.ap-northeast-1.compute.internal : ok=559 changed=99 unreachable=0 failed=0 skipped=889 rescued=0 ignored=1
ip-10-250-199-180.ap-northeast-1.compute.internal : ok=515 changed=96 unreachable=0 failed=0 skipped=771 rescued=0 ignored=1
ip-10-250-207-144.ap-northeast-1.compute.internal : ok=646 changed=138 unreachable=0 failed=0 skipped=1100 rescued=0 ignored=3
ip-10-250-210-49.ap-northeast-1.compute.internal : ok=515 changed=96 unreachable=0 failed=0 skipped=771 rescued=0 ignored=1
ip-10-250-218-202.ap-northeast-1.compute.internal : ok=515 changed=96 unreachable=0 failed=0 skipped=771 rescued=0 ignored=1
ip-10-250-221-144.ap-northeast-1.compute.internal : ok=644 changed=137 unreachable=0 failed=0 skipped=1102 rescued=0 ignored=3
localhost : ok=3 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
Monday 21 August 2023 14:50:01 +0900 (0:00:00.159) 0:22:02.315 *********
===============================================================================
kubernetes/preinstall : Install packages requirements ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 221.47s
download : Download_file | Download item --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 64.53s
download : Download_file | Download item --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 33.97s
download : Download_file | Download item --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 25.71s
kubernetes/kubeadm : Join to cluster ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 24.37s
kubernetes/control-plane : Joining control plane node to the cluster. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 21.13s
download : Download_container | Download image if required --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 18.37s
kubespray-defaults : Gather ansible_default_ipv4 from all hosts ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 18.16s
etcd : Gen_certs | Write etcd member/admin and kube_control_plane client certs to other etcd nodes ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 18.08s
download : Download_container | Download image if required --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 17.93s
kubernetes/control-plane : Kubeadm | Initialize first master ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 14.27s
bootstrap-os : Update Apt cache ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 13.34s
download : Download_container | Download image if required --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 12.60s
container-engine/containerd : Download_file | Download item -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 12.51s
container-engine/crictl : Download_file | Download item ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 11.94s
container-engine/runc : Download_file | Download item -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 11.57s
container-engine/crictl : Extract_file | Unpacking archive --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 11.53s
container-engine/nerdctl : Download_file | Download item ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 11.43s
container-engine/containerd : Containerd | Unpack containerd archive ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 11.33s
container-engine/nerdctl : Extract_file | Unpacking archive --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 9.96s
アクセス確認
まず、MasterNodeからアクセスできるかを確認
MasterNodeのいずれかに接続する
inventory/hostsか、AWSコンソールでMasterNodeタグが付いているものがそれにあたる。
ProxyCommandにbastionサーバを指定し、MasterNodeまでSSH接続する。
ssh -o ProxyCommand='ssh -i ~/.ssh/k8s.pem -W %h:%p admin@3.113.30.102' -i ~/.ssh/k8s.pem admin@10.250.197.20
rootに昇格
sudo su -
kubectlをたたくことができた
root@ip-10-250-221-144:~# kubectl get nodes
NAME STATUS ROLES AGE VERSION
ip-10-250-197-20.ap-northeast-1.compute.internal Ready control-plane 29m v1.27.4
ip-10-250-198-1.ap-northeast-1.compute.internal Ready <none> 27m v1.27.4
ip-10-250-199-180.ap-northeast-1.compute.internal Ready <none> 27m v1.27.4
ip-10-250-207-144.ap-northeast-1.compute.internal Ready control-plane 28m v1.27.4
ip-10-250-210-49.ap-northeast-1.compute.internal Ready <none> 27m v1.27.4
ip-10-250-218-202.ap-northeast-1.compute.internal Ready <none> 27m v1.27.4
ip-10-250-221-144.ap-northeast-1.compute.internal Ready control-plane 28m v1.27.4
ローカルマシンから確認する
マスターノードの~/.kube/configをコピーし、clusters.cluster.serverの箇所をELBのFQDNに書き換えて、ローカルに保存する
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURCVENDQWUyZ0F3SUJBZ0lJVFdRUE5RUFhWUUF3RFFZSktvWklodmNOQVFFTEJRQXdGVEVUTUJFR0ExVUUKQXhNS2EzVmlaWEp1WlhSbGN6QWVGdzB5TXpBNE1qRXdOVFExTlROYUZ3MHpNekE0TVRnd05UUTFOVE5hTUJVeApFekFSQmdOVkJBTVRDbXQxWW1WeWJtVjBaWE13Z2dFaU1BMEdDU3FHU0liM0RRRUJBUVVBQTRJQkR3QXdnZ0VLCkFvSUJBUUMxOXgxYWplaDh0d0NaM3J1SCtCR0xrYzNFbUswNjZWZDhjZmRaQmhhbk4rbE9vWmd5WFVXOTVtRHYKTUF3RUllNFBWYVRwV0Jjb3V5dktxWXIvRGdqMVh1OWE2SE9JMGs3cXdpR29NUGJZVXlNYUJYd3d0NFRRMk9RbApTT2Vwd2FFZmx1OU55TGdhWEx2cmJFRW1JUk8zUktpZXNvdTQ2UW51dXNuVnhkZ1o0WkxvdVJONjh1WTMyVHl0ClMzMHh0OHZmMExFOUFWWWlkdGpPZllZbU8wR1pRSXpUQWV6N1Jwek1ZQ0E5RUJ3VjFkU04zVVJOK3p5aGV3QUEKblEwU1hsZFNYVnljQnZCR2VZOVVOeGEzdjdocll3U25uc0lZT0ZzanpwTnpYSlNsbElPM2NZdkRManBEMmJldQpza1FzdUtZQU9ROHRsak8zZ3IraG9ITEVhVW9UQWdNQkFBR2pXVEJYTUE0R0ExVWREd0VCL3dRRUF3SUNwREFQCkJnTlZIUk1CQWY4RUJUQURBUUgvTUIwR0ExVWREZ1FXQkJSRWVVL1FTdEt1VnFrZTUyRHllaTd1SVcyOXhqQVYKQmdOVkhSRUVEakFNZ2dwcmRXSmxjbTVsZEdWek1BMEdDU3FHU0liM0RRRUJDd1VBQTRJQkFRQnFwSWJ2QTRMTQpadFd6WEpBWjU5NHVUYXBIUWIyWEtINERuSG13YVAyUXZMN3dPZFgyNE43VkxTVWxaTEpZUXN3eEN2R055Zk54CnNoUEZjSmk2bFZ6dHF5RklGa21oQ2hsbGpTRFlhK1IwQ2w2cmhaMGRTV3lBdThLaXQyNlhXY3FrWkZabmFoT3QKMTc1VmtVK2NrK2ZLUTlncS96c1A4S0Z0Zlo5Q3RsN0kzVWZmcW5jTzg2c3BrRXhBbUx1US93TUNkTzB6Y3VPaApTNXZ0MFNYblRBV3RhR1VtbGRPL0wwNFhadS94ZjBzZFgrOVZNR0p5TEp5b2s4Tnp2dy9XdHJ2NXptU050ODd3ClUwM2s3a05Sd2o3b0dtVGpuSXpYZ09ZeUJlOXJTOGluU1Y4Rmd6TzlSOHZxeGl5MlFaZGljV0tXMS9BTC81RFMKeXpnamhlWWhlL0RRCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
server: https://kubernetes-nlb-devtest-035904b132ec747b.elb.ap-northeast-1.amazonaws.com:6443
name: cluster.local
contexts:
- context:
cluster: cluster.local
user: kubernetes-admin
name: kubernetes-admin@cluster.local
current-context: kubernetes-admin@cluster.local
kind: Config
preferences: {}
users:
- name: kubernetes-admin
user:
client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURJVENDQWdtZ0F3SUJBZ0lJUkcvU2RyMENDSlF3RFFZSktvWklodmNOQVFFTEJRQXdGVEVUTUJFR0ExVUUKQXhNS2EzVmlaWEp1WlhSbGN6QWVGdzB5TXpBNE1qRXdOVFExTlROYUZ3MHlOREE0TWpBd05UUTJNak5hTURReApGekFWQmdOVkJBb1REbk41YzNSbGJUcHRZWE4wWlhKek1Sa3dGd1lEVlFRREV4QnJkV0psY201bGRHVnpMV0ZrCmJXbHVNSUlCSWpBTkJna3Foa2lHOXcwQkFRRUZBQU9DQVE4QU1JSUJDZ0tDQVFFQXlLdGJYRmp3VmZEdlNOeVYKOENrcXQ5d1kveU9ydUx6aGV4dHJUdFRTakx2RDBPVVk1WnViWEc1d3lTLzVOT0JZVzFmTDZXOVNRdm5LY0ZqVgplcGZ2dURRbCtscVg4UGxnbmpyM3JCWFNqU2laRFZaQUZIRzVwMng4dWk2cFN3MFQvU21TaEZvbldaZTBUektIClJTcVVFbHp5dnVOcVcvcUlSc3lSM3Y1TW9ySE5sUGM0R29pMjdNVWdObG1WZTlhV05yU256VzdZZHFFWkNDcnoKOFhZRjNVVGZaUXVaUDAxOW9Kbk94VmNGWmdlRmV5Z0p3Q2xRSGxmRXZkYks3RUkyRVdWZ1k4UjZ1VEh0aGZBRQpFUzg0OTM4Y3dCajJZdUZUNzhQRmpqUDd5Z29LV0xJUGxSV3JrZkhBVmN5OVhSbG5tRXVMZTNzTkdJOWlLeU9aCmR2UGpyd0lEQVFBQm8xWXdWREFPQmdOVkhROEJBZjhFQkFNQ0JhQXdFd1lEVlIwbEJBd3dDZ1lJS3dZQkJRVUgKQXdJd0RBWURWUjBUQVFIL0JBSXdBREFmQmdOVkhTTUVHREFXZ0JSRWVVL1FTdEt1VnFrZTUyRHllaTd1SVcyOQp4akFOQmdrcWhraUc5dzBCQVFzRkFBT0NBUUVBZmNuWk9hS041a2g5Q2ZvMFJMZi9PYzFVSGdLL1B0NmF1cC8zCjc0RlhTM3ozaXkycmJaemo4R2xOalJ5RWlNYm5aeDQ2VkpqclVIZUxqRS9OSXNhdzFGdUJyVnpDZUUwKzB0MCsKbkxRQklOUjViM2NMUkFmd2xMTGQzVU0yd2JFODJwUTA5VDEzdXNpeWtqUjd3Rkk1QmpEL3RwdTBkVWZ5VHZ5YQowTTVhaytWVU9BTE5QL0h4Zmh5L2xCOEtuWHllUkNsMzB3S09WQVkyWlpNNlR4U1U1c3M5OHp3T1plSmhCMkg4Cmo3SS9UL2JkYnA5b0pZOGdFL1EvQm05cEY5bnk5aEZjZzJvR3pDTUpiVUZMK0JLWHB4bGJ3VGNNdGtZck1mUncKaU5weTRmZVVHeDRFRnYranMvNFlVb3NqOUJqTE9RR09Xd1dEMzFiYlZlMmd0OWNuUFE9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
client-key-data: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcEFJQkFBS0NBUUVBeUt0YlhGandWZkR2U055VjhDa3F0OXdZL3lPcnVMemhleHRyVHRUU2pMdkQwT1VZCjVadWJYRzV3eVMvNU5PQllXMWZMNlc5U1F2bktjRmpWZXBmdnVEUWwrbHFYOFBsZ25qcjNyQlhTalNpWkRWWkEKRkhHNXAyeDh1aTZwU3cwVC9TbVNoRm9uV1plMFR6S0hSU3FVRWx6eXZ1TnFXL3FJUnN5UjN2NU1vckhObFBjNApHb2kyN01VZ05sbVZlOWFXTnJTbnpXN1lkcUVaQ0NyejhYWUYzVVRmWlF1WlAwMTlvSm5PeFZjRlpnZUZleWdKCndDbFFIbGZFdmRiSzdFSTJFV1ZnWThSNnVUSHRoZkFFRVM4NDkzOGN3QmoyWXVGVDc4UEZqalA3eWdvS1dMSVAKbFJXcmtmSEFWY3k5WFJsbm1FdUxlM3NOR0k5aUt5T1pkdlBqcndJREFRQUJBb0lCQUI1NmVGdmtYa3lQVmFpNQphRVNiM0JNazZOOEhxZk5iTE53bjQvcUxOSmVQaFVSYko1Ym9OWWFxZDVMM0tHQ3htbm4xTWRGNHlRUEhNTW96Cmsra1UzcUladVpMMjQ1RUVMM3FZOWprNjcwbmxXcW05NmVteEJFTzdwSTRhdkRmYitxYy9BelA3QlhNRUthekYKdmVueGQ1LzZZaHhBS2sxaUZnUzYyWVVIRnRpMjUrSXV0dDR4a1J5OHRwM0xiUDRYOE1PZ0I3dDdZSXJBenExbwpFRDlab2dlUGtjVzNvOEpuNkt5T1I3TVg3UmxQN0JRa1BMbHVmSWEyUmdnS3kzTExrdmhhOGNzTjFCYTRoeHNpCjlJWElpNWY2UlpiNDJyUmVlQ2dPeTZyWkphSW00Sjcrc0VLTHVDNWZScXFaZzdoSHArSExYUFFPMlBQUGJxZFQKNlNsdVVpRUNnWUVBNmVkWStNb2t2ZmJENitFRXRiTExCeFdZUzVsZVF6VS9TNUZOTSszNXlxWi9SRjI3dFVDagplVEE4NXUxdFlBTkdpWjdxVUtPdHNYNHkveHc2emg4cXJoNExmTXNRaEFhRVo1amIrR05Wem5jVys3NFB5VlVmCnZyaG8rS0pFQyswaFVOeXdDL1hkV3dRM0ZDY0pMR0V1ZFM3MTJEL2NNZk1qbWh4N2Q4Yzg5MWtDZ1lFQTI2QkgKMjdXYlNPSzRjREZGL1IzWEVscjAzNVhFQzNRUVYzYVhDSnNTY1RFU241TEZsMDAzWTRpRGVmMnRRU2ZIL3JBTQpod1ZBKzZUQW14WElpM0lGMThneXNNYjBTQTU1NzhxUUVXdkc5MTk5by93cElGNXVGbloramxVYXF4dzI0MmlWCkpGcjIwVEc1R2svSUZhVDBXd3d6MWVxQ2wyTWlrME9FNG9vR1drY0NnWUVBM2VYQTdUendBRnRDNE5wMjJsMEUKNlBLZXFWR1ZNRk1xZWVrZGdaOUk4dXhqSGh1Q1FPQlNkVnpDMmh6eVB6eThSVURDYjcvbnZ0UEhVMmVYamdYWgpPcGQ4TkdaOGc1T2ZOcHk0U2ZzSzR6VHRVK09xSEVrcU5ZZkM1ZVVpZ3VIeVVrRWFyaTRRVWhFOG8xeXBBallyCm1iWDBPclBTbEh4aUVRam1ST3A1dHZrQ2dZRUF5WHgxOEI3ZjR6SDhPeEU1UUVtTDhnNzM3OWNhRDlmZTFoNjUKMk85TE1BVHhMT3dkMGZ6SS84dFJzeTBETVc4cGo0STdPL09aOWtMallHUFBvLzVyVmRjWWlnMDhObzd4VndmaQp1OWpBcEZrNmpxTGJxQko1dEtubzJQTkNrQkJHV3NLcmNMcnpER0F0alBIOGdiN2ZYbGdIaGd5cTJlL2MzSFZwCkI5dlFrQ2tDZ1lCK2xVUWhXTTlKZTJ4MWZSV0tEeG9ZR3kvMUREWXVqUHZhRjlvVUJIVGlvKzdkQk95WjZ3MFcKajZad2lIemRxaXAxdmgxK1g2bmFaV3c5TUVRMHUwbW9rWWpCWENZbHVHeTJYSUdtNm9VVkFUYzlKNEtxUDN0TgpHM0JtcFhtNDRjMmhmNDFyOTQwQjdwUkxYakQxaDRqZFkwcnJWTHF1cHZka3ZIUTBwSktTc3c9PQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=
kubeconfigを読み込む
export KUBECONFIG=~/.kube/config.kubesplay
kubectlがローカルから実行できた
yuta master ~ kubespray kubectl get nodes NAME STATUS ROLES AGE VERSION
ip-10-250-197-20.ap-northeast-1.compute.internal Ready control-plane 45m v1.27.4
ip-10-250-198-1.ap-northeast-1.compute.internal Ready <none> 44m v1.27.4
ip-10-250-199-180.ap-northeast-1.compute.internal Ready <none> 44m v1.27.4
ip-10-250-207-144.ap-northeast-1.compute.internal Ready control-plane 45m v1.27.4
ip-10-250-210-49.ap-northeast-1.compute.internal Ready <none> 44m v1.27.4
ip-10-250-218-202.ap-northeast-1.compute.internal Ready <none> 44m v1.27.4
ip-10-250-221-144.ap-northeast-1.compute.internal Ready control-plane 45m v1.27.4
アンインストールコマンド
ansible-playbook -i ./inventory/hosts ./reset.yml -e ansible_user=admin -b --become-user=root --flush-cache
Discussion