iTranslated by AI
My Notes on Trying Out Cybozu's 'Introduction to Kubernetes' New Employee Training
Introduction
Training materials for new employees at Cybozu have been made public, and among them is a training module titled "Introduction to Kubernetes". This post serves as a personal memo of my experience trying out that module as a Kubernetes beginner.
The original site provides very detailed explanations, so please refer to it if you are looking for in-depth information. Also, I won't be covering much about kubectl commands, but this article was very easy to understand.
Impressions
My impression after trying it out is that it is very easy to follow, and I would recommend it for anyone looking to get started with Kubernetes.
Before and after going through this training, my level of understanding when reading other Kubernetes-related articles and slides is completely different, so it was highly beneficial.
About the Training Materials
The materials are composed of the following three parts:
- Introduction to Kubernetes
- More Introduction to Kubernetes
- Exercise
"Introduction to Kubernetes" is the main entry-level section, while "More Introduction to Kubernetes" covers slightly more advanced content, such as "things you need to consider for production operations." "Exercise" consists of practice problems, which I did not attempt (my apologies).
This article will provide a comprehensive overview of "Introduction to Kubernetes."
I will only briefly touch upon the contents of "More Introduction to Kubernetes."
Introduction to Kubernetes
This is the basics section.
Setting up the Kubernetes Environment
We will use Minikube.
Minikube allows you to start a single-node Kubernetes cluster.
It seems that multi-node Kubernetes clusters cannot be created with Minikube.
Additionally, to use Minikube, you need to set CPUs: 2 or more in your Docker settings.
(I was warned and Minikube would not start with CPUs: 1.)
Displaying the Kubernetes Dashboard
# Install Minikube
$ brew install minikube
# Check version
$ minikube version
minikube version: v1.12.3
commit: 2243b4b97c131e3244c5f014faedca0d846599f5
# Check status
$ minikube status
minikube
type: Control Plane
host: Running
kubelet: Running
apiserver: Running
kubeconfig: Configured
# Start dashboard
$ minikube dashboard
The dashboard will appear as shown below.
Deploying a Container on Kubernetes
The smallest unit that can be deployed on Kubernetes is a Pod.
A Pod is a collection of one or more containers. Containers belonging to the same Pod are always deployed on the same node.
Conceptually, the hierarchy is Container < Pod < Node.
By the way, Minikube acts as the Node.
Prepare a YAML file configured for the Pod to be deployed on Kubernetes.
For now, we will write it with the following content.
apiVersion: v1
kind: Pod
metadata:
name: my-first-pod
labels:
component: nginx
spec:
containers:
- name: nginx
image: nginx:latest
Deploy to Kubernetes using the kubectl apply -f [YAML file name to apply] command.
# Apply to Kubernetes
$ kubectl apply -f nginx-pod.yaml
pod/my-first-pod created
# List Pods in the cluster
$ kubectl get pod
NAME READY STATUS RESTARTS AGE
my-first-pod 1/1 Running 0 103s
# Display details of the Pod
$ kubectl describe pod my-first-pod
Name: my-first-pod
Namespace: default
Priority: 0
Node: minikube/172.17.0.3
Start Time: Sat, 15 Aug 2020 04:15:37 +0900
Labels: component=nginx
（Omitted）
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 5m5s default-scheduler Successfully assigned default/my-first-pod to minikube
Normal Pulling 5m4s kubelet, minikube Pulling image "nginx:latest"
Normal Pulled 4m50s kubelet, minikube Successfully pulled image "nginx:latest"
Normal Created 4m50s kubelet, minikube Created container nginx
Normal Started 4m50s kubelet, minikube Started container nginx
# Login to the Pod
kubectl exec -it my-first-pod -- /bin/bash
# Install curl command
root@my-first-pod:/# apt update && apt install -y curl
（Omitted）
# Access localhost:80 (Check connectivity)
root@my-first-pod:/# curl -i localhost:80
HTTP/1.1 200 OK
（Omitted）
my-first-pod is displayed in the dashboard as follows.
Accessing Other Pods
Check connectivity between Pods.
Prepare a Pod different from my-first-pod.
apiVersion: v1
kind: Pod
metadata:
name: bastion
spec:
containers:
- name: bastion
image: debian:stretch
command: ["sleep", "infinity"]
By the way, "bastion" means a "jump host."
Check connectivity from bastion to my-first-pod (nginx).
# Apply to Kubernetes
$ kubectl apply -f bastion.yaml
pod/bastion created
# List Pods
$ kubectl get pod
NAME READY STATUS RESTARTS AGE
bastion 1/1 Running 0 14s
my-first-pod 1/1 Running 1 10h
# Display IP addresses
$ kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
bastion 1/1 Running 0 84s 172.18.0.6 minikube <none> <none>
my-first-pod 1/1 Running 1 10h 172.18.0.2 minikube <none> <none>
# Login to bastion and access 172.18.0.2 (the IP address of my-first-pod)
$ kubectl exec -it bastion -- bash
(bastion)# apt update && apt install -y curl
(bastion)# curl -i http://172.18.0.2/
HTTP/1.1 200 OK
（Omitted）
The dashboard confirms that bastion has been added.
Accessing Other Pods Using a Service
We just tested Pod-to-Pod connectivity, but since Pods disappear and are recreated, their IP addresses change, so it's not ideal to communicate directly with Pod IP addresses.
Therefore, we want a stable (= non-changing) endpoint to communicate with "any Pod that provides the desired function" rather than a specific Pod.
Usually, when communicating with other Pods, we use an object called a Service. Using a Service allows you to access "one of the Pods with a specific label" instead of "a specific Pod."
In the nginx example above, the label part is component: nginx. The words "component" and "nginx" do not have special meanings here; you can specify any string you like.
This time, we will use the following Service.
apiVersion: v1
kind: Service
metadata:
name: my-first-service
spec:
selector:
component: nginx
ports:
- protocol: TCP
port: 80
targetPort: 80
Apply the Service to Kubernetes.
The method for applying is the same as for Pods.
$ kubectl apply -f nginx-service.yaml
service/my-first-service created
$ kubectl get service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 12h
my-first-service ClusterIP 10.101.46.86 <none> 80/TCP 3m7s
The dashboard confirms that my-first-service has been added.
Now, let's access my-first-pod (labeled component: nginx) from the bastion Pod through my-first-service, just like before.
$ kubectl exec -it bastion -- bash
root@bastion:/# curl -i http://my-first-service/
HTTP/1.1 200 OK
（Omitted）
It can be illustrated as follows:
Launching Multiple Instances of the Same Pod
In the previous section, we launched one nginx instance. In actual production environments, it is common to configure a single service with multiple Pods for redundancy and scalability. Here, we will use the Kubernetes ReplicaSet object to launch the specified number of nginx instances.
Below is the ReplicaSet definition we will use this time.
apiVersion: apps/v1
kind: ReplicaSet
metadata:
name: nginx-replicaset
labels:
component: nginx
spec:
replicas: 3
selector:
matchLabels:
component: nginx
template:
metadata:
labels:
component: nginx
spec:
containers:
- name: nginx
image: nginx:latest
While I will omit the detailed explanation, ReplicaSet is an object that automatically deploys or deletes Pods so that the "number of Pods matching the selector" equals the "number specified in replicas."
For now, let's apply it and check the number of pods.
$ kubectl get pod
NAME READY STATUS RESTARTS AGE
bastion 1/1 Running 0 109m
my-first-pod 1/1 Running 1 12h
$ kubectl apply -f nginx-replicaset.yaml
replicaset.apps/nginx-replicaset created
$ kubectl get replicaset
NAME DESIRED CURRENT READY AGE
nginx-replicaset 3 3 3 13s
$ kubectl get pod
NAME READY STATUS RESTARTS AGE
bastion 1/1 Running 0 110m
my-first-pod 1/1 Running 1 12h
nginx-replicaset-wqcdk 1/1 Running 0 28s
nginx-replicaset-xbs28 1/1 Running 0 28s
Looking at the dashboard, you can confirm that there are 3 Pods running.
Now, let's try deleting one of the Pods.
Even after deleting a Pod, it will immediately launch again so that there are 3 Pods.
$ kubectl delete pod my-first-pod
pod "my-first-pod" deleted
$ kubectl get pod
NAME READY STATUS RESTARTS AGE
bastion 1/1 Running 0 123m
nginx-replicaset-dn57j 1/1 Running 0 9s
nginx-replicaset-wqcdk 1/1 Running 0 13m
nginx-replicaset-xbs28 1/1 Running 0 13m
In the dashboard, you can confirm that there are still 3 Pods running.
Next, let's change replicas: 3 to replicas: 4 and apply the changes.
$ kubectl apply -f nginx-replicaset.yaml
replicaset.apps/nginx-replicaset configured
$ kubectl get pod
NAME READY STATUS RESTARTS AGE
bastion 1/1 Running 0 126m
nginx-replicaset-5swkc 1/1 Running 0 7s
nginx-replicaset-dn57j 1/1 Running 0 3m33s
nginx-replicaset-wqcdk 1/1 Running 0 16m
nginx-replicaset-xbs28 1/1 Running 0 16m
There are now 4 nginx Pods.
Of course, you can also confirm this on the dashboard.
Finally, let's delete the ReplicaSet from this state.
$ kubectl delete replicaset nginx-replicaset
replicaset.apps "nginx-replicaset" deleted
$ kubectl get pod
NAME READY STATUS RESTARTS AGE
bastion 1/1 Running 0 163m
nginx-replicaset-5swkc 0/1 Terminating 0 37m
nginx-replicaset-dn57j 0/1 Terminating 0 40m
nginx-replicaset-wqcdk 0/1 Terminating 0 54m
nginx-replicaset-xbs28 0/1 Terminating 0 54m
The nginx Pods are now gone.
Checking the dashboard, you can confirm that both the nginx Pods and the ReplicaSet have been removed.
Rolling Updates
To perform rolling updates in Kubernetes, you use an object called Deployment. A Deployment is an object similar to a ReplicaSet, but unlike a ReplicaSet, it supports updates.
Incidentally, if you perform rolling updates with ReplicaSet without using Deployment, you need to follow a procedure like the one shown in the figure below.
With Deployment, you can have Kubernetes handle this complex procedure for you.
The YAML for the Deployment used this time is as follows. It is almost the same as the ReplicaSet.
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
component: nginx
spec:
replicas: 3
selector:
matchLabels:
component: nginx
template:
metadata:
labels:
component: nginx
spec:
containers:
- name: nginx
image: nginx:1.15
Let's try applying the Deployment.
$ kubectl apply -f nginx-deployment.yaml
deployment.apps/nginx-deployment created
$ kubectl get deployment
NAME READY UP-TO-DATE AVAILABLE AGE
nginx-deployment 3/3 3 3 40s
# Verification (I am displaying the IMAGE to check the version)
$ kubectl get pod -o 'custom-columns=NAME:.metadata.name,IMAGE:.spec.containers[*].image,PHASE:.status.phase'
NAME IMAGE PHASE
bastion debian:stretch Running
nginx-deployment-77bc96745b-9tdr2 nginx:1.15 Running
nginx-deployment-77bc96745b-kg96q nginx:1.15 Running
nginx-deployment-77bc96745b-xn4v6 nginx:1.15 Running
It is displayed like this on the dashboard.
Now, let's change image: nginx:1.15 to image: nginx:1.16 and apply the Deployment.
$ kubectl apply -f nginx-deployment.yaml
deployment.apps/nginx-deployment configured
If you continue to run the kubectl get pod command from this point on, you can observe Kubernetes performing the rolling update.
$ kubectl get pod -o 'custom-columns=NAME:.metadata.name,IMAGE:.spec.containers[*].image,PHASE:.status.phase'
NAME IMAGE PHASE
bastion debian:stretch Running
nginx-deployment-5b4c7f657-pnwkr nginx:1.16 Pending
nginx-deployment-77bc96745b-9tdr2 nginx:1.15 Running
nginx-deployment-77bc96745b-kg96q nginx:1.15 Running
nginx-deployment-77bc96745b-xn4v6 nginx:1.15 Running
$ kubectl get pod -o 'custom-columns=NAME:.metadata.name,IMAGE:.spec.containers[*].image,PHASE:.status.phase'
NAME IMAGE PHASE
bastion debian:stretch Running
nginx-deployment-5b4c7f657-dcqwd nginx:1.16 Running
nginx-deployment-5b4c7f657-pnwkr nginx:1.16 Running
nginx-deployment-5b4c7f657-t9ddv nginx:1.16 Running
nginx-deployment-77bc96745b-9tdr2 nginx:1.15 Running
nginx-deployment-77bc96745b-kg96q nginx:1.15 Running
nginx-deployment-77bc96745b-xn4v6 nginx:1.15 Running
$ kubectl get pod -o 'custom-columns=NAME:.metadata.name,IMAGE:.spec.containers[*].image,PHASE:.status.phase'
NAME IMAGE PHASE
bastion debian:stretch Running
nginx-deployment-5b4c7f657-dcqwd nginx:1.16 Running
nginx-deployment-5b4c7f657-pnwkr nginx:1.16 Running
nginx-deployment-5b4c7f657-t9ddv nginx:1.16 Running
As you can see, all Pods have transitioned from image: nginx:1.15 to image: nginx:1.16.
You can also verify this on the dashboard.
It is said that when actually using Kubernetes, you rarely create Pods or ReplicaSets directly.
The best practice in most cases is to use Deployment when creating Pods.
Even if the number of replicas is 1, you should use Deployment to create Pods. The reason is simply that if you deploy a single Pod, if the node where the Pod is running dies, the Pod will also be lost.
Exposing Services Outside the Cluster
I will explain how to expose a service deployed on Kubernetes to the outside of the Kubernetes cluster. There are several ways to expose a service, but here I will explain the easiest method using NodePort.
Change the Service as follows:
apiVersion: v1
kind: Service
metadata:
name: my-first-service
spec:
selector:
component: nginx
ports:
- protocol: TCP
port: 80
targetPort: 80
nodePort: 30000
type: NodePort
The lines nodePort: 30000 and type: NodePort have been added.
Apply the changes.
$ kubectl get service my-first-service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
my-first-service ClusterIP 10.101.46.86 <none> 80/TCP 3m7s
$ kubectl apply -f nginx-service.yaml
service/my-first-service configured
# The TYPE has changed
$ kubectl get service my-first-service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
my-first-service NodePort 10.101.46.86 <none> 80:30000/TCP 124m
Through this operation, the type of my-first-service has been changed from ClusterIP (default) to NodePort.
You can also confirm that port 30000 is open on the dashboard.
From this point on, it differs from the training material: accessing via <IP address returned by minikube ip : 30000> did not work, so I executed it using the command below, which seems to expose the IP address and port number.
$ minikube service my-first-service --url
🏃 Starting tunnel for service my-first-service.
|-----------|------------------|-------------|------------------------|
| NAMESPACE | NAME | TARGET PORT | URL |
|-----------|------------------|-------------|------------------------|
| default | my-first-service | | http://127.0.0.1:53157 |
|-----------|------------------|-------------|------------------------|
http://127.0.0.1:53157
When accessing http://127.0.0.1:53157 from a browser, I was able to connect to nginx as shown below.
Debugging Pods
In many cases, you need to debug deployed Pods.
kubectl exec for logging in and kubectl describe are useful for debugging Pods.
In this section, I will explain two other useful debugging commands in addition to those two: kubectl logs and kubectl port-forward.
Viewing Container Logs
You can check logs using the kubectl logs command.
# Current list of Pods
$ kubectl get pod
NAME READY STATUS RESTARTS AGE
nginx-deployment-5b4c7f657-cbd8r 1/1 Running 0 103m
nginx-deployment-5b4c7f657-fvd88 1/1 Running 0 103m
nginx-deployment-5b4c7f657-lqqkm 1/1 Running 0 103m
# Display logs for a specific Pod
$ kubectl logs nginx-deployment-5b4c7f657-cbd8r
172.17.0.3 - - [15/Aug/2020:10:32:24 +0000] "GET / HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36" "-"
172.17.0.3 - - [15/Aug/2020:10:32:24 +0000] "GET /favicon.ico HTTP/1.1" 404 555 "http://127.0.0.1:52378/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36" "-"
2020/08/15 10:32:24 [error] 6#6: *1 open() "/usr/share/nginx/html/favicon.ico" failed (2: No such file or directory), client: 172.17.0.3, server: localhost, request: "GET /favicon.ico HTTP/1.1", host: "127.0.0.1:52378", referrer: "http://127.0.0.1:52378/"
# Display logs for Pods matching a label
$ kubectl logs -l component=nginx
172.17.0.3 - - [15/Aug/2020:10:32:24 +0000] "GET / HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36" "-"
172.17.0.3 - - [15/Aug/2020:10:32:24 +0000] "GET /favicon.ico HTTP/1.1" 404 555 "http://127.0.0.1:52378/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36" "-"
2020/08/15 10:32:24 [error] 6#6: *1 open() "/usr/share/nginx/html/favicon.ico" failed (2: No such file or directory), client: 172.17.0.3, server: localhost, request: "GET /favicon.ico HTTP/1.1", host: "127.0.0.1:52378", referrer: "http://127.0.0.1:52378/"
172.17.0.3 - - [15/Aug/2020:10:45:24 +0000] "GET / HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36" "-"
2020/08/15 10:45:25 [error] 6#6: *1 open() "/usr/share/nginx/html/favicon.ico" failed (2: No such file or directory), client: 172.17.0.3, server: localhost, request: "GET /favicon.ico HTTP/1.1", host: "127.0.0.1:52724", referrer: "http://127.0.0.1:52724/"
172.17.0.3 - - [15/Aug/2020:10:45:25 +0000] "GET /favicon.ico HTTP/1.1" 404 555 "http://127.0.0.1:52724/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36" "-"
172.17.0.3 - - [15/Aug/2020:12:01:44 +0000] "GET / HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36" "-"
2020/08/15 12:01:44 [error] 6#6: *3 open() "/usr/share/nginx/html/favicon.ico" failed (2: No such file or directory), client: 172.17.0.3, server: localhost, request: "GET /favicon.ico HTTP/1.1", host: "127.0.0.1:53157", referrer: "http://127.0.0.1:53157/"
172.17.0.3 - - [15/Aug/2020:12:01:44 +0000] "GET /favicon.ico HTTP/1.1" 404 555 "http://127.0.0.1:53157/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36" "-"
Port Forwarding
Using the kubectl port-forward command, you can forward a specific port of a Pod to localhost.
Using this, you can access the Pod directly from a local curl command or browser.
For example, if you run the following command, it will forward port 8080 locally to port 80 of the nginx container in the cluster.
$ kubectl port-forward deployment/nginx-deployment 8080:80
When you access localhost:8080, "Welcome to nginx!" will be displayed.
More Introduction to Kubernetes
I will explain important features assuming operation in a production environment.
Here, I will just touch on them briefly.
Health Checks
Kubernetes has the following two health checks, which can be used according to the purpose:
- Liveness probe: Checks whether a Pod is alive or dead. If the Liveness probe fails a certain number of times, the Pod will be restarted.
- Readiness probe: Checks whether a Pod can respond to requests. Until the Readiness probe succeeds, the Pod is considered not to have finished starting up and is not added to the service's reverse proxy target. Using a Readiness probe can prevent requests from being sent before the Pod has finished starting up.
Since I could not run the sample from the training material, I defined the YAML file by referring to this article.
apiVersion: v1
kind: Pod
metadata:
name: sample-healthcheck
labels:
app: sample-app
spec:
containers:
- name: nginx-container
image: nginx:1.12
ports:
- containerPort: 80
livenessProbe:
httpGet:
path: /index.html
port: 80
scheme: HTTP
timeoutSeconds: 1
successThreshold: 1
failureThreshold: 2
initialDelaySeconds: 5
periodSeconds: 3
readinessProbe:
exec:
command: ["ls", "/usr/share/nginx/html/50x.html"]
timeoutSeconds: 1
successThreshold: 2
failureThreshold: 1
initialDelaySeconds: 5
periodSeconds: 3
Let's apply it.
$ kubectl apply -f sample-healthcheck.yaml
pod/sample-healthcheck created
$ kubectl get pod
NAME READY STATUS RESTARTS AGE
sample-healthcheck 1/1 Running 0 17s
# Verify Liveness settings with describe
$ kubectl describe pod sample-healthcheck | grep "Liveness"
Liveness: http-get http://:80/index.html delay=5s timeout=1s period=3s #success=1 #failure=2
# Verify Readiness settings with describe
$ kubectl describe pod sample-healthcheck | grep "Readiness"
Readiness: exec [ls /usr/share/nginx/html/50x.html] delay=5s timeout=1s period=3s #success=2 #failure=1
I will experiment by intentionally triggering the health check.
# First, monitor the Pod status in a separate console
$ kubectl get pods sample-healthcheck --watch
# Delete index.html in DocumentRoot (deleting the file monitored by Liveness)
$ kubectl exec -it sample-healthcheck rm /usr/share/nginx/html/index.html
# The Pod restarts (since index.html is created after restarting, it will not repeat the restart)
$ kubectl get pods sample-healthcheck --watch
NAME READY STATUS RESTARTS AGE
sample-healthcheck 0/1 Running 2 9m12s
sample-healthcheck 1/1 Running 2 9m20s
Liveness probes and Readiness probes are very important for operations, so it is recommended to always set both for resident Pods.
Resource Requests/Limits
You can pre-configure resources for Pods.
There are two types of settings:
- Resource Requests: A feature to declare the amount of memory or CPU required by the container. The Kubernetes scheduler decides the node to deploy the Pod based on the Resource Requests value.
- Resource Limits: A feature to set the upper limit for the amount of memory or CPU that the container can actually use. If the memory used by the container exceeds the set limit, the container is killed. Also, if CPU usage reaches the limit, the container is throttled.
Here is an example of a YAML file.
apiVersion: v1
kind: Pod
metadata:
name: nginx
labels:
component: nginx
spec:
containers:
- name: nginx
image: nginx:1.16
resources:
requests:
memory: "128Mi"
cpu: "250m"
limits:
memory: "128Mi"
cpu: "500m"
Apply and check the settings.
$ kubectl apply -f sample-resource.yaml
$ kubectl describe pod nginx
(omitted)
Limits:
cpu: 500m
memory: 128Mi
Requests:
cpu: 250m
memory: 128Mi
(omitted)
Volume
When a Pod has multiple containers, you may want to share a local directory between them.
For example, there is a use case where log files output by an AP server are forwarded by a log forwarding agent.
By the way, the sample uses fluentd, which is quite famous as an OSS data log collection tool.
apiVersion: v1
kind: Pod
metadata:
name: nginx
labels:
component: nginx
spec:
containers:
- name: nginx
image: nginx:1.16
volumeMounts:
- name: nginx-log
mountPath: /var/log/nginx
- name: fluentd
image: fluent/fluentd:v1.11
volumeMounts:
- name: nginx-log
mountPath: /var/log/nginx
readOnly: true
volumes:
- name: nginx-log
emptyDir: {}
Apply and check.
$ kubectl apply -f sample-volume.yaml
Pod/nginx created
$ kubectl get pod
NAME READY STATUS RESTARTS AGE
nginx 2/2 Running 0 2m5s
$ kubectl describe pod nginx
(omitted)
Containers:
nginx:
Container ID: docker://3ebb817cf4ee20439c722bbeba711943bb5f05a2d75b420d308faaf1edbd0af2
Image: nginx:1.16
Image ID: docker-pullable://nginx@sha256:d20aa6d1cae56fd17cd458f4807e0de462caf2336f0b70b5eeb69fcaaf30dd9c
Port: <none>
Host Port: <none>
State: Running
Started: Sat, 15 Aug 2020 22:26:10 +0900
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/log/nginx from nginx-log (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-q8zvm (ro)
fluentd:
Container ID: docker://f001646ac161052c67e6aba83ec999b97cb55650c4db095c310e60af75a871cc
Image: fluent/fluentd:v1.11
Image ID: docker-pullable://fluent/fluentd@sha256:617fa61a8a811fd49b730c27835983ecb2a8150584138ec825649ff8352a6d44
Port: <none>
Host Port: <none>
State: Running
Started: Sat, 15 Aug 2020 22:28:13 +0900
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/log/nginx from nginx-log (ro)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-q8zvm (ro)
(omitted)
Conceptually, the diagram looks like this.
Anti Affinity
Suppose you are using Deployment to create two identical Pods. If you don't specify anything, those two Pods might be scheduled on the same node. In such a case, if that node dies due to failure or maintenance, both Pods will disappear simultaneously.
Using the Affinity feature, you can request the scheduler to place Pods on different nodes. Kubernetes Affinity is very flexible and can do various things, but here I will focus on the use case mentioned above.
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
component: nginx
spec:
replicas: 2
selector:
matchLabels:
component: nginx
template:
metadata:
labels:
component: nginx
spec:
containers:
- name: nginx
image: nginx:1.16
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: component
operator: In
values:
- "nginx"
topologyKey: "kubernetes.io/hostname"
In the definition, I set replicas: 2 so that two Pods will be created.
With the description starting from affinity:, when Kubernetes deploys this Pod, it tries to deploy it to a node that does not have a Pod with the label component: nginx.
This ensures that even if one node dies, at least one nginx Pod survives.
Let's apply it in the single-node environment of Minikube.
$ kubectl apply -f sample-affinity.yaml
deployment.apps/nginx-deployment created
$ kubectl get pod
NAME READY STATUS RESTARTS AGE
nginx-deployment-6b7b554df-9p4b9 0/1 Pending 0 71s
nginx-deployment-6b7b554df-xf5bp 1/1 Running 0 71s
Because it is a single-node, you can see that once one Pod is started, the other Pod is not deployed and is in a Pending state.
Like this, when there is no node that satisfies the condition, the Pod is not deployed and stays in a Pending state until a node that satisfies the condition appears.
It is recommended to include Pods that have been Pending for a long time in your monitoring items.
Pod Disruption Budget
There may be cases where node reboots are necessary. For example, when a vulnerability is discovered in the Linux kernel.
If you set up multiple Pods and make them redundant, you can withstand the shutdown of a single node. However, problems can occur in situations where nodes are rebooted one after another.
For example, consider the case where two Pods are deployed on a cluster consisting of three nodes as shown in the figure below. These two Pods are configured to have 2 replicas by Deployment. We will perform a rolling reboot of the nodes one by one on this cluster.
In the process of shutting down the first node, the Pod running on it is Evicted. The Deployment, which detects that the number of Pods has decreased, immediately deploys a new Pod. However, since it takes time for this Pod to start, it stays in the starting state. In this situation, if you shut down node 2, no available Pods will exist, and the service will stop.
To prevent this, use Pod Disruption Budget to specify the minimum number of Pods required for the service to run.
When PDB is specified, Kubernetes tools wait appropriately so that this constraint is satisfied before shutting down a node.
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
name: nginx-pdb
spec:
minAvailable: 1
selector:
matchLabels:
component: nginx
Apply and check.
$ kubectl apply -f frontend-pdb.yaml
poddisruptionbudget.policy/nginx-pdb created
$ kubectl get poddisruptionbudget
NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGE
nginx-pdb 1 N/A 0 66s
Naturally, the PDB constraints are observed only when the cluster administrator intentionally shuts down a node. If a node dies due to failure, PDB is ignored. Also, PDB is not applied when deleting a Deployment.
Whether a Pod is available is determined by the Readiness probe. To make PDB function correctly, you must set appropriate Readiness probes for your Pods.
Exercise
I have not done the Exercise in the training material.
Sorry...
That's all.
Discussion