🚀
kubeadmでEpehemeral Containerを有効にする
kubernetes 1.22で kubectl debug コマンドを有効にするには --feature-gates で有効化をする必要があります．
k8s1% kubectl debug sample-pod --image=amsy810/tools:v2.0 -it -- bash
Defaulting debug container name to debugger-lc6mh.
error: ephemeral containers are disabled for this cluster (error from server: "the server could not find the requested resource").
環境は次の通り．
% kubeadm config images list
k8s.gcr.io/kube-apiserver:v1.22.1
k8s.gcr.io/kube-controller-manager:v1.22.1
k8s.gcr.io/kube-scheduler:v1.22.1
k8s.gcr.io/kube-proxy:v1.22.1
k8s.gcr.io/pause:3.5
k8s.gcr.io/etcd:3.5.0-0
k8s.gcr.io/coredns/coredns:v1.8.4
TL;DR
コンポーネントのConfigで --feature-gates=EphemeralContainers=true をControl plane， worker nodeの 全ノードで有効化 する必要があります．
Control plane
% sudo vim /var/lib/kubelet/kubeadm-flags.env
KUBELET_KUBEADM_ARGS="--container-runtime=remote --container-runtime-endpoint=/run/containerd/containerd.sock --pod-infra-container-image=k8s.gcr.io/pause:3.4.1 --feature-gates=EphemeralContainers=true"
% sudo vim /etc/kubernetes/manifests/kube-apiserver.yaml
apiVersion: v1
kind: Pod
...
- --etcd-servers=https://127.0.0.1:2379
- --feature-gates=EphemeralContainers=true
- --insecure-port=0
...
% sudo vim /etc/kubernetes/manifests/kube-scheduler.yaml
apiVersion: v1
kind: Pod
...
- --bind-address=127.0.0.1
- --feature-gates=EphemeralContainers=true
- --kubeconfig=/etc/kubernetes/scheduler.conf
% sudo systemctl daemon-reload
% sudo systemctl restart kubelet
Worker nodes
% sudo vim /var/lib/kubelet/kubeadm-flags.env
KUBELET_KUBEADM_ARGS="--container-runtime=remote --container-runtime-endpoint=/run/containerd/containerd.sock --pod-infra-container-image=k8s.gcr.io/pause:3.4.1 --feature-gates=EphemeralContainers=true"
% sudo systemctl daemon-reload
% sudo systemctl restart kubelet
Ephemeral ContainerはAlpha
v1.22時点ではデフォルト無効化されており，明示的に有効化が必要です．
Feature Default Stage Since Until
EphemeralContainers false Alpha 1.16
kubeadmの設定
how to enable resource EphemeralContainers in Rancher kubernetes
で設定例がありました．
最初はControl planeのコンポーネントだけに設定していましたが，worker nodesで動くkubeletコンポーネントにも設定することを忘れていました．
Control plane
% sudo vim /var/lib/kubelet/kubeadm-flags.env
KUBELET_KUBEADM_ARGS="--container-runtime=remote --container-runtime-endpoint=/run/containerd/containerd.sock --pod-infra-container-image=k8s.gcr.io/pause:3.4.1 --feature-gates=EphemeralContainers=true"
% sudo vim /etc/kubernetes/manifests/kube-apiserver.yaml
apiVersion: v1
kind: Pod
...
- --etcd-servers=https://127.0.0.1:2379
- --feature-gates=EphemeralContainers=true
- --insecure-port=0
...
% sudo vim /etc/kubernetes/manifests/kube-scheduler.yaml
apiVersion: v1
kind: Pod
...
- --bind-address=127.0.0.1
- --feature-gates=EphemeralContainers=true
- --kubeconfig=/etc/kubernetes/scheduler.conf
Worker nodes
% sudo vim /var/lib/kubelet/kubeadm-flags.env
KUBELET_KUBEADM_ARGS="--container-runtime=remote --container-runtime-endpoint=/run/containerd/containerd.sock --pod-infra-container-image=k8s.gcr.io/pause:3.4.1 --feature-gates=EphemeralContainers=true"
% sudo systemctl daemon-reload
% sudo systemctl restart kubelet
kubectl debug
Debug Running Pods に従って試すと，シェルが取れるようになっています．
k8s1% kubectl run ephemeral-demo --image=k8s.gcr.io/pause:3.1 --restart=Never
pod/ephemeral-demo created
k8s1% kubectl exec -it ephemeral-demo -- sh
error: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "b3495e0398f0597e479e534be958ed8f254afc0e45052162cfb13847b3bee79a": OCI runtime exec failed: exec failed: container_linux.go:380: starting container process caused: exec: "sh": executable file not found in $PATH: unknown
k8s1% kubectl debug -it ephemeral-demo --image=busybox --target=ephemeral-demo
Targeting container "ephemeral-demo". If you don't see processes from this container it may be because the container runtime doesn't support this feature.
Defaulting debug container name to debugger-pgvbp.
If you don't see a command prompt, try pressing enter.
/ # hostname
ephemeral-demo
Epehmeral Containerが起動していることがわかります．
k8s1% k describe po ephemeral-demo
Name: ephemeral-demo
Namespace: default
Priority: 0
Node: k8s3/192.168.5.112
Start Time: Mon, 30 Aug 2021 22:01:23 +0900
Labels: run=ephemeral-demo
Annotations: cni.projectcalico.org/podIP: 10.244.219.11/32
cni.projectcalico.org/podIPs: 10.244.219.11/32
Status: Running
IP: 10.244.219.11
IPs:
IP: 10.244.219.11
Containers:
ephemeral-demo:
Container ID: containerd://1710d670716c4c480168cf1d76a6554739e591a00a99eed5f91fd783a8f526c7
Image: k8s.gcr.io/pause:3.1
Image ID: k8s.gcr.io/pause@sha256:f78411e19d84a252e53bff71a4407a5686c46983a2c2eeed83929b888179acea
Port: <none>
Host Port: <none>
State: Running
Started: Mon, 30 Aug 2021 22:01:23 +0900
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-nv77s (ro)
Ephemeral Containers:
debugger-pgvbp:
Container ID: containerd://ca669ac90de7852367c1977eef322d5388778e9d95db5a4c71d39f82523aa981
Image: busybox
Image ID: docker.io/library/busybox@sha256:b37dd066f59a4961024cf4bed74cae5e68ac26b48807292bd12198afa3ecb778
Port: <none>
Host Port: <none>
State: Terminated
Reason: Completed
Exit Code: 0
Started: Mon, 30 Aug 2021 22:01:37 +0900
Finished: Mon, 30 Aug 2021 22:02:34 +0900
Ready: False
Restart Count: 0
Environment: <none>
Mounts: <none>
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
kube-api-access-nv77s:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 83s default-scheduler Successfully assigned default/ephemeral-demo to k8s3
Normal Pulled 83s kubelet Container image "k8s.gcr.io/pause:3.1" already present on machine
Normal Created 83s kubelet Created container ephemeral-demo
Normal Started 83s kubelet Started container ephemeral-demo
Normal Pulling 71s kubelet Pulling image "busybox"
Normal Pulled 69s kubelet Successfully pulled image "busybox" in 1.755345903s
Normal Created 69s kubelet Created container debugger-pgvbp
Normal Started 69s kubelet Started container debugger-pgvbp
ref
Discussion