Closed1ヶ月前にクローズ6

K3s の Kubernetes クラスタを破壊した時の復旧備忘録

finalizer 関連で引っかかるので、External Secrets Operator の CRD とか ClusterRole をまとめて削除。

> kubectl get crd | grep "external-secrets.io" | awk '{print $1}' | xargs kubectl delete crd
customresourcedefinition.apiextensions.k8s.io "acraccesstokens.generators.external-secrets.io" deleted
customresourcedefinition.apiextensions.k8s.io "clusterexternalsecrets.external-secrets.io" deleted
customresourcedefinition.apiextensions.k8s.io "clustergenerators.generators.external-secrets.io" deleted
customresourcedefinition.apiextensions.k8s.io "clusterpushsecrets.external-secrets.io" deleted
customresourcedefinition.apiextensions.k8s.io "clustersecretstores.external-secrets.io" deleted
customresourcedefinition.apiextensions.k8s.io "ecrauthorizationtokens.generators.external-secrets.io" deleted
customresourcedefinition.apiextensions.k8s.io "externalsecrets.external-secrets.io" deleted
customresourcedefinition.apiextensions.k8s.io "fakes.generators.external-secrets.io" deleted
customresourcedefinition.apiextensions.k8s.io "gcraccesstokens.generators.external-secrets.io" deleted
customresourcedefinition.apiextensions.k8s.io "generatorstates.generators.external-secrets.io" deleted
customresourcedefinition.apiextensions.k8s.io "githubaccesstokens.generators.external-secrets.io" deleted
customresourcedefinition.apiextensions.k8s.io "grafanas.generators.external-secrets.io" deleted
customresourcedefinition.apiextensions.k8s.io "mfas.generators.external-secrets.io" deleted
customresourcedefinition.apiextensions.k8s.io "passwords.generators.external-secrets.io" deleted
customresourcedefinition.apiextensions.k8s.io "pushsecrets.external-secrets.io" deleted
customresourcedefinition.apiextensions.k8s.io "quayaccesstokens.generators.external-secrets.io" deleted
customresourcedefinition.apiextensions.k8s.io "secretstores.external-secrets.io" deleted
customresourcedefinition.apiextensions.k8s.io "stssessiontokens.generators.external-secrets.io" deleted
customresourcedefinition.apiextensions.k8s.io "uuids.generators.external-secrets.io" deleted
customresourcedefinition.apiextensions.k8s.io "vaultdynamicsecrets.generators.external-secrets.io" deleted
customresourcedefinition.apiextensions.k8s.io "webhooks.generators.external-secrets.io" deleted

> kubectl get ClusterRole | grep "external-secrets" | awk '{print $1}' | xargs kubectl delete ClusterRole
clusterrole.rbac.authorization.k8s.io "external-secrets-operator-controller" deleted
clusterrole.rbac.authorization.k8s.io "external-secrets-operator-edit" deleted
clusterrole.rbac.authorization.k8s.io "external-secrets-operator-servicebindings" deleted
clusterrole.rbac.authorization.k8s.io "external-secrets-operator-view" deleted

> kubectl get ClusterRoleBinding | grep "external-secrets" | awk '{print $1}' | xargs kubectl delete ClusterRoleBinding
clusterrolebinding.rbac.authorization.k8s.io "external-secrets-operator-cert-controller" deleted
clusterrolebinding.rbac.authorization.k8s.io "external-secrets-operator-controller" deleted

これで argocd Namespace が削除できるようになったので、Namespace 定義のファイルなどでまとめて削除。

残る Namespace は以下。

> k get ns
NAME              STATUS   AGE
default           Active   87d
kube-node-lease   Active   87d
kube-public       Active   87d
kube-system       Active   87d

そこから Namespace を再作成。

> k apply -f apps/manifests/namespaces.yaml
namespace/argocd created
namespace/cert-manager created
namespace/cloudflare-tunnel created
namespace/cnpg-system created
namespace/country-roads created
namespace/external-secrets-operator created
namespace/forgejo created
namespace/gitlab-runner created
namespace/ipinfo created
namespace/penpot created
namespace/monitoring created
namespace/nenga-oct216 created
namespace/speedtest created
namespace/system-upgrade created
namespace/tailscale created
namespace/upsnap created
namespace/uptime-kuma created

External Secrets Operator で利用する Access Token を GitLab Project (※) で発行。

https://gitlab.com/oct216/argocd-apps/-/settings/access_tokens
- ※ CI/CD 変数を Secret の管理に利用しているため

k3s-external-secrets という名前にしてた。

>  cat <<'EOF' | kubectl apply -f - -n external-secrets-operator
pipe heredoc> apiVersion: v1
kind: Secret
metadata:
  name: gitlab-secret
  labels:
    type: gitlab
type: Opaque
stringData:
  token: "**access token goes here**"
pipe heredoc> EOF
secret/gitlab-secret created

External Secrets Operator を導入。

helmfile -q template とかのオプションは Argo CD の CMP から持ってきた。

>  cd ./applicationset/sync-1st/external-secrets-operator

> helmfile -q template --include-crds --skip-tests | k apply -f -
serviceaccount/external-secrets-cert-controller created
serviceaccount/external-secrets-operator created
serviceaccount/external-secrets-webhook created
secret/external-secrets-operator-webhook created
customresourcedefinition.apiextensions.k8s.io/acraccesstokens.generators.external-secrets.io created
customresourcedefinition.apiextensions.k8s.io/clusterexternalsecrets.external-secrets.io created
customresourcedefinition.apiextensions.k8s.io/clustergenerators.generators.external-secrets.io created
customresourcedefinition.apiextensions.k8s.io/clusterpushsecrets.external-secrets.io created
customresourcedefinition.apiextensions.k8s.io/clustersecretstores.external-secrets.io created
customresourcedefinition.apiextensions.k8s.io/ecrauthorizationtokens.generators.external-secrets.io created
customresourcedefinition.apiextensions.k8s.io/externalsecrets.external-secrets.io created
customresourcedefinition.apiextensions.k8s.io/fakes.generators.external-secrets.io created
customresourcedefinition.apiextensions.k8s.io/gcraccesstokens.generators.external-secrets.io created
customresourcedefinition.apiextensions.k8s.io/generatorstates.generators.external-secrets.io created
customresourcedefinition.apiextensions.k8s.io/githubaccesstokens.generators.external-secrets.io created
customresourcedefinition.apiextensions.k8s.io/grafanas.generators.external-secrets.io created
customresourcedefinition.apiextensions.k8s.io/mfas.generators.external-secrets.io created
customresourcedefinition.apiextensions.k8s.io/passwords.generators.external-secrets.io created
customresourcedefinition.apiextensions.k8s.io/pushsecrets.external-secrets.io created
customresourcedefinition.apiextensions.k8s.io/quayaccesstokens.generators.external-secrets.io created
customresourcedefinition.apiextensions.k8s.io/secretstores.external-secrets.io created
customresourcedefinition.apiextensions.k8s.io/stssessiontokens.generators.external-secrets.io created
customresourcedefinition.apiextensions.k8s.io/uuids.generators.external-secrets.io created
customresourcedefinition.apiextensions.k8s.io/vaultdynamicsecrets.generators.external-secrets.io created
customresourcedefinition.apiextensions.k8s.io/webhooks.generators.external-secrets.io created
clusterrole.rbac.authorization.k8s.io/external-secrets-operator-cert-controller created
clusterrole.rbac.authorization.k8s.io/external-secrets-operator-controller created
clusterrole.rbac.authorization.k8s.io/external-secrets-operator-view created
clusterrole.rbac.authorization.k8s.io/external-secrets-operator-edit created
clusterrole.rbac.authorization.k8s.io/external-secrets-operator-servicebindings created
clusterrolebinding.rbac.authorization.k8s.io/external-secrets-operator-cert-controller created
clusterrolebinding.rbac.authorization.k8s.io/external-secrets-operator-controller created
role.rbac.authorization.k8s.io/external-secrets-operator-leaderelection created
rolebinding.rbac.authorization.k8s.io/external-secrets-operator-leaderelection created
service/external-secrets-operator-webhook created
deployment.apps/external-secrets-operator-cert-controller created
deployment.apps/external-secrets-operator created
deployment.apps/external-secrets-operator-webhook created
validatingwebhookconfiguration.admissionregistration.k8s.io/secretstore-validate unchanged
validatingwebhookconfiguration.admissionregistration.k8s.io/externalsecret-validate unchanged

ClusterSecretStore もまとめて apply したいところだけれど、ESO の apply 直後は以下のようなエラーが出る。

Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "validate.clustersecretstore.external-secrets.io": failed to call webhook: Post "https://external-secrets-operator-webhook.external-secrets-operator.svc:443/validate-external-secrets-io-v1-clustersecretstore?timeout=5s": no endpoints available for service "external-secrets-operator-webhook"

3 分ぐらい時間をおいてから再 apply すると、作成される。

clustersecretstore.external-secrets.io/gitlab-secret-store created

Argo CD の導入。

> helmfile -q template --include-crds --skip-tests | k apply -f - -n argocd
externalsecret.external-secrets.io/gitlab-application-credentials created
externalsecret.external-secrets.io/gitlab-private-repo created
serviceaccount/argocd-application-controller created
serviceaccount/argocd-applicationset-controller created
serviceaccount/argocd-notifications-controller created
serviceaccount/argocd-repo-server created
serviceaccount/argocd-server created
secret/argocd-notifications-secret created
secret/argocd-secret created
configmap/argocd-cm created
configmap/argocd-cmd-params-cm created
configmap/argocd-cmp-cm created
configmap/argocd-gpg-keys-cm created
configmap/argocd-notifications-cm created
configmap/argocd-rbac-cm created
configmap/argocd-ssh-known-hosts-cm created
configmap/argocd-tls-certs-cm created
configmap/argocd-redis-health-configmap created
customresourcedefinition.apiextensions.k8s.io/applications.argoproj.io created
customresourcedefinition.apiextensions.k8s.io/applicationsets.argoproj.io created
customresourcedefinition.apiextensions.k8s.io/appprojects.argoproj.io created
clusterrole.rbac.authorization.k8s.io/argocd-application-controller created
clusterrole.rbac.authorization.k8s.io/argocd-notifications-controller created
clusterrole.rbac.authorization.k8s.io/argocd-server created
clusterrolebinding.rbac.authorization.k8s.io/argocd-application-controller created
clusterrolebinding.rbac.authorization.k8s.io/argocd-notifications-controller created
clusterrolebinding.rbac.authorization.k8s.io/argocd-server created
role.rbac.authorization.k8s.io/argocd-application-controller created
role.rbac.authorization.k8s.io/argocd-applicationset-controller created
role.rbac.authorization.k8s.io/argocd-notifications-controller created
role.rbac.authorization.k8s.io/argocd-repo-server created
role.rbac.authorization.k8s.io/argocd-server created
rolebinding.rbac.authorization.k8s.io/argocd-application-controller created
rolebinding.rbac.authorization.k8s.io/argocd-applicationset-controller created
rolebinding.rbac.authorization.k8s.io/argocd-notifications-controller created
rolebinding.rbac.authorization.k8s.io/argocd-repo-server created
rolebinding.rbac.authorization.k8s.io/argocd-server created
service/argocd-applicationset-controller created
service/argocd-repo-server created
service/argocd-server created
service/argocd-redis created
deployment.apps/argocd-applicationset-controller created
deployment.apps/argocd-notifications-controller created
deployment.apps/argocd-repo-server created
deployment.apps/argocd-server created
deployment.apps/argocd-redis created
statefulset.apps/argocd-application-controller created
serviceaccount/argocd-redis-secret-init created
role.rbac.authorization.k8s.io/argocd-redis-secret-init created
rolebinding.rbac.authorization.k8s.io/argocd-redis-secret-init created
job.batch/argocd-redis-secret-init created

Cloudflare Tunnel の導入。

命名とか気になるところは残っているので、後日修正予定。

> helmfile -q template --include-crds --skip-tests | k apply -f - -n cloudflare-tunnel
externalsecret.external-secrets.io/ipinfo-cloudflare-tunnel-remote created
externalsecret.external-secrets.io/speedtest-cloudflare-tunnel-remote created
externalsecret.external-secrets.io/countryroads-cloudflare-tunnel-remote created
externalsecret.external-secrets.io/argocd-cloudflare-tunnel-remote created
serviceaccount/ipinfo-cloudflare-tunnel-remote created
deployment.apps/ipinfo-cloudflare-tunnel-remote created
serviceaccount/speedtest-cloudflare-tunnel-remote created
deployment.apps/speedtest-cloudflare-tunnel-remote created
serviceaccount/countryroads-cloudflare-tunnel-remote created
deployment.apps/countryroads-cloudflare-tunnel-remote created
serviceaccount/argocd-cloudflare-tunnel-remote created
deployment.apps/argocd-cloudflare-tunnel-remote created

Cloudflare Zero Trust 側の設定は済んでいたので、アクセスするだけで利用できる。

Argo CD Application のデプロイ。
> helmfile -q template --include-crds --skip-tests | k apply -f -
namespace/argocd configured
namespace/cert-manager configured
namespace/cloudflare-tunnel configured
namespace/cnpg-system configured
namespace/country-roads configured
namespace/external-secrets-operator configured
namespace/forgejo configured
namespace/gitlab-runner configured
namespace/ipinfo configured
namespace/penpot configured
namespace/monitoring configured
namespace/nenga-oct216 configured
namespace/speedtest configured
namespace/system-upgrade configured
namespace/tailscale configured
namespace/upsnap configured
namespace/uptime-kuma configured
application.argoproj.io/apps created
applicationset.argoproj.io/applicationset-sync-1st created
application.argoproj.io/cloudnative-pg created
application.argoproj.io/system-upgrade-controller created
applicationset.argoproj.io/applicationset-sync-2nd created
application.argoproj.io/k3s-upgrade-plans created
application.argoproj.io/plugin-barman-cloud created
applicationset.argoproj.io/applicationset-sync-3rd created
順番に Sync されている。
全部 Healthy になるまで 5〜8 分ほどかかる。
※ ゲストアクセスでは権限を絞っているため Destination などは Unknown として表示される。

このスクラップは1ヶ月前にクローズされました