Closed6
RKEで構築したオンプレKubernetesクラスタにIngress設定
yumenomatayume- 前提条件
- オンプレ環境
- metallbがデプロイされている
- metallbからアサインしたIPに設定
- RKEでデフォルトでインストールされているingressを使用
- Rancher Docs: K8s Ingress Controllers
-
default-http-backendというserviceがある
- オンプレ環境
yumenomatayume以下のパッチを当ててtype: LoadBalancerに変更します。
kubectl patch svc default-http-backend -n ingress-nginx -p '{"spec": {"type": "LoadBalancer"}}'
EXTERNAL-IPにmetallbのセグメントのIPアドレスが割り振られていることを確認します。
$ kubectl get svc -n ingress-nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default-http-backend LoadBalancer 10.43.201.185 192.168.100.129 80:32270/TCP 14h
$ kubectl describe svc -n ingress-nginx
Name: default-http-backend
Namespace: ingress-nginx
Labels: app=default-http-backend
Annotations: <none>
Selector: app=default-http-backend
Type: LoadBalancer
IP Families: <none>
IP: 10.43.201.185
IPs: 10.43.201.185
LoadBalancer Ingress: 192.168.100.129
Port: <unset> 80/TCP
TargetPort: 8080/TCP
NodePort: <unset> 32270/TCP
Endpoints: 10.42.0.4:8080
Session Affinity: None
External Traffic Policy: Cluster
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal nodeAssigned 113s (x9482 over 14h) metallb-speaker announcing from node "192.168.100.4"
実際にHTTPアクセスしてみます。
$ curl 192.168.100.129
default backend - 404
上記によるとingress-controllerの設定を調べる必要があります。
RKEの場合は名前が変わっているようなので、以下で確認します。。
$ kubectl get configmap -n kube-system
NAME DATA AGE
coredns 1 14h
coredns-autoscaler 1 14h
extension-apiserver-authentication 6 14h
full-cluster-state 1 14h
kube-flannel-cfg 2 14h
kube-root-ca.crt 1 14h
rke-coredns-addon 1 14h
rke-ingress-controller 1 14h
rke-metrics-addon 1 14h
rke-network-plugin 1 14h
rke-ingress-controllerを見てみます。
$ kubectl get configmap -n kube-system rke-ingress-controller -o yaml
apiVersion: v1
data:
rke-ingress-controller: "\n# this template is intended for use by >= nginx-0.32.0\napiVersion:
v1\nkind: Namespace\nmetadata:\n name: ingress-nginx\n---\nkind: ConfigMap\napiVersion:
v1\nmetadata:\n name: nginx-configuration\n namespace: ingress-nginx\n labels:\n
\ app: ingress-nginx\ndata:\n\n---\nkind: ConfigMap\napiVersion: v1\nmetadata:\n
\ name: tcp-services\n namespace: ingress-nginx\n---\nkind: ConfigMap\napiVersion:
v1\nmetadata:\n name: udp-services\n namespace: ingress-nginx\n\n---\napiVersion:
v1\nkind: ServiceAccount\nmetadata:\n name: nginx-ingress-serviceaccount\n namespace:
ingress-nginx\n---\napiVersion: rbac.authorization.k8s.io/v1beta1\nkind: ClusterRole\nmetadata:\n
\ name: nginx-ingress-clusterrole\nrules:\n - apiGroups:\n - \"\"\n resources:\n
\ - configmaps\n - endpoints\n - nodes\n - pods\n - secrets\n
\ verbs:\n - list\n - watch\n - apiGroups:\n - \"\"\n resources:\n
\ - nodes\n verbs:\n - get\n - apiGroups:\n - \"\"\n resources:\n
\ - services\n verbs:\n - get\n - list\n - update\n -
watch\n - apiGroups:\n - extensions\n - \"networking.k8s.io\" # k8s
1.14+\n resources:\n - ingresses\n verbs:\n - get\n - list\n
\ - watch\n - apiGroups:\n - \"\"\n resources:\n - events\n
\ verbs:\n - create\n - patch\n - apiGroups:\n - extensions\n
\ - \"networking.k8s.io\" # k8s 1.14+\n resources:\n - ingresses/status\n
\ verbs:\n - update\n---\napiVersion: rbac.authorization.k8s.io/v1beta1\nkind:
Role\nmetadata:\n name: nginx-ingress-role\n namespace: ingress-nginx\nrules:\n
\ - apiGroups:\n - \"\"\n resources:\n - namespaces\n verbs:\n
\ - get\n - apiGroups:\n - \"\"\n resources:\n - configmaps\n
\ - pods\n - secrets\n - endpoints\n verbs:\n - get\n -
list\n - watch\n - apiGroups:\n - \"\"\n resources:\n - services\n
\ verbs:\n - get\n - list\n - update\n - watch\n - apiGroups:\n
\ - extensions\n - \"networking.k8s.io\" # k8s 1.14+\n resources:\n
\ - ingresses\n verbs:\n - get\n - list\n - watch\n - apiGroups:\n
\ - extensions\n - \"networking.k8s.io\" # k8s 1.14+\n resources:\n
\ - ingresses/status\n verbs:\n - update\n - apiGroups:\n -
\"\"\n resources:\n - configmaps\n resourceNames:\n # Defaults
to \"<election-id>-<ingress-class>\"\n # Here: \"<ingress-controller-leader>-<nginx>\"\n
\ # This has to be adapted if you change either parameter\n # when launching
the nginx-ingress-controller.\n - ingress-controller-leader-nginx\n verbs:\n
\ - get\n - update\n - apiGroups:\n - \"\"\n resources:\n -
configmaps\n verbs:\n - create\n - apiGroups:\n - \"\"\n resources:\n
\ - endpoints\n verbs:\n - create\n - get\n - update\n -
apiGroups:\n - \"\"\n resources:\n - events\n verbs:\n -
create\n - patch\n---\napiVersion: rbac.authorization.k8s.io/v1beta1\nkind:
RoleBinding\nmetadata:\n name: nginx-ingress-role-nisa-binding\n namespace:
ingress-nginx\nroleRef:\n apiGroup: rbac.authorization.k8s.io\n kind: Role\n
\ name: nginx-ingress-role\nsubjects:\n - kind: ServiceAccount\n name: nginx-ingress-serviceaccount\n
\ namespace: ingress-nginx\n---\napiVersion: rbac.authorization.k8s.io/v1beta1\nkind:
ClusterRoleBinding\nmetadata:\n name: nginx-ingress-clusterrole-nisa-binding\nroleRef:\n
\ apiGroup: rbac.authorization.k8s.io\n kind: ClusterRole\n name: nginx-ingress-clusterrole\nsubjects:\n
\ - kind: ServiceAccount\n name: nginx-ingress-serviceaccount\n namespace:
ingress-nginx\n\n---\napiVersion: apps/v1\nkind: DaemonSet\nmetadata:\n name:
nginx-ingress-controller\n namespace: ingress-nginx\nspec:\n selector:\n matchLabels:\n
\ app: ingress-nginx\n\n updateStrategy:\n rollingUpdate:\n maxUnavailable:
1\n type: RollingUpdate\n\n template:\n metadata:\n labels:\n app:
ingress-nginx\n annotations:\n prometheus.io/port: '10254'\n prometheus.io/scrape:
'true'\n spec:\n affinity:\n nodeAffinity:\n requiredDuringSchedulingIgnoredDuringExecution:\n
\ nodeSelectorTerms:\n - matchExpressions:\n -
key: beta.kubernetes.io/os\n operator: NotIn\n values:\n
\ - windows\n - key: node-role.kubernetes.io/worker\n
\ operator: Exists\n hostNetwork: true\n \n# Rancher
specific change\n\n \n serviceAccountName: nginx-ingress-serviceaccount\n
\ \n terminationGracePeriodSeconds: 60\n tolerations:\n - effect:
NoExecute\n operator: Exists\n - effect: NoSchedule\n operator:
Exists\n containers:\n - name: nginx-ingress-controller\n image:
rancher/nginx-ingress-controller:nginx-0.43.0-rancher3\n args:\n -
/nginx-ingress-controller\n - --default-backend-service=$(POD_NAMESPACE)/default-http-backend\n
\ - --configmap=$(POD_NAMESPACE)/nginx-configuration\n -
--election-id=ingress-controller-leader\n - --ingress-class=nginx\n
\ - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services\n -
--udp-services-configmap=$(POD_NAMESPACE)/udp-services\n - --annotations-prefix=nginx.ingress.kubernetes.io\n
\ \n securityContext:\n capabilities:\n drop:\n
\ - ALL\n add:\n - NET_BIND_SERVICE\n
\ runAsUser: 101\n env:\n - name: POD_NAME\n valueFrom:\n
\ fieldRef:\n fieldPath: metadata.name\n -
name: POD_NAMESPACE\n valueFrom:\n fieldRef:\n fieldPath:
metadata.namespace\n\n ports:\n - name: http\n containerPort:
80\n - name: https\n containerPort: 443\n livenessProbe:\n
\ failureThreshold: 3\n httpGet:\n path: /healthz\n
\ port: 10254\n scheme: HTTP\n initialDelaySeconds:
10\n periodSeconds: 10\n successThreshold: 1\n timeoutSeconds:
1\n readinessProbe:\n failureThreshold: 3\n httpGet:\n
\ path: /healthz\n port: 10254\n scheme:
HTTP\n initialDelaySeconds: 10\n periodSeconds: 10\n successThreshold:
1\n timeoutSeconds: 1\n\n\n---\napiVersion: apps/v1\nkind: Deployment\nmetadata:\n
\ name: default-http-backend\n labels:\n app: default-http-backend\n namespace:
ingress-nginx\nspec:\n replicas: 1\n selector:\n matchLabels:\n app:
default-http-backend\n template:\n metadata:\n labels:\n app:
default-http-backend\n spec:\n affinity:\n nodeAffinity:\n requiredDuringSchedulingIgnoredDuringExecution:\n
\ nodeSelectorTerms:\n - matchExpressions:\n -
key: beta.kubernetes.io/os\n operator: NotIn\n values:\n
\ - windows\n - key: node-role.kubernetes.io/worker\n
\ operator: Exists\n terminationGracePeriodSeconds: 60\n#
Rancher specific change\n tolerations:\n - effect: NoExecute\n operator:
Exists\n - effect: NoSchedule\n operator: Exists\n containers:\n
\ - name: default-http-backend\n # Any image is permissable as long
as:\n # 1. It serves a 404 page at /\n # 2. It serves 200 on a /healthz
endpoint\n image: rancher/mirrored-nginx-ingress-controller-defaultbackend:1.5-rancher1\n
\ livenessProbe:\n httpGet:\n path: /healthz\n port:
8080\n scheme: HTTP\n initialDelaySeconds: 30\n timeoutSeconds:
5\n ports:\n - containerPort: 8080\n resources:\n limits:\n
\ cpu: 10m\n memory: 20Mi\n requests:\n cpu:
10m\n memory: 20Mi\n---\napiVersion: v1\nkind: Service\nmetadata:\n
\ name: default-http-backend\n namespace: ingress-nginx\n labels:\n app:
default-http-backend\nspec:\n ports:\n - port: 80\n targetPort: 8080\n selector:\n
\ app: default-http-backend\n"
kind: ConfigMap
metadata:
creationTimestamp: "2021-07-24T16:08:08Z"
name: rke-ingress-controller
namespace: kube-system
resourceVersion: "583"
uid: 722e18f7-133f-49f4-a36f-209ddb196c92
そもそもhttp2: falseという項目はなさそうでした。
yumenomatayume
上記にRKEでのセットアップ方法が公開されていたので、試してみます。
metallbはデプロイ完了しているので、nginxからデプロイしてみます。
git clone https://gitlab.com/monachus/channel.git
cd resources/2020.01/deploy/2-nginx
applyしてみます。
$ kubectl apply -k .
namespace/ingress-nginx configured
serviceaccount/ingress-nginx created
serviceaccount/ingress-nginx-admission created
role.rbac.authorization.k8s.io/ingress-nginx created
role.rbac.authorization.k8s.io/ingress-nginx-admission created
clusterrole.rbac.authorization.k8s.io/ingress-nginx created
clusterrole.rbac.authorization.k8s.io/ingress-nginx-admission created
rolebinding.rbac.authorization.k8s.io/ingress-nginx created
rolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
configmap/ingress-nginx-controller created
configmap/tcp-services configured
configmap/udp-services configured
service/ingress-nginx-controller created
service/ingress-nginx-controller-admission created
deployment.apps/ingress-nginx-controller created
job.batch/ingress-nginx-admission-create created
job.batch/ingress-nginx-admission-patch created
validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission created
作成されたリソースは以下になっています。
$ kubectl get -k .
NAME STATUS AGE
namespace/ingress-nginx Active 16h
NAME SECRETS AGE
serviceaccount/ingress-nginx 1 50s
serviceaccount/ingress-nginx-admission 1 50s
NAME CREATED AT
role.rbac.authorization.k8s.io/ingress-nginx 2021-07-25T08:20:34Z
role.rbac.authorization.k8s.io/ingress-nginx-admission 2021-07-25T08:20:34Z
NAME CREATED AT
clusterrole.rbac.authorization.k8s.io/ingress-nginx 2021-07-25T08:20:34Z
clusterrole.rbac.authorization.k8s.io/ingress-nginx-admission 2021-07-25T08:20:34Z
NAME ROLE AGE
rolebinding.rbac.authorization.k8s.io/ingress-nginx Role/ingress-nginx 50s
rolebinding.rbac.authorization.k8s.io/ingress-nginx-admission Role/ingress-nginx-admission 50s
NAME ROLE AGE
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx ClusterRole/ingress-nginx 50s
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx-admission ClusterRole/ingress-nginx-admission 50s
NAME DATA AGE
configmap/ingress-nginx-controller 0 50s
configmap/tcp-services 0 16h
configmap/udp-services 0 16h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/ingress-nginx-controller LoadBalancer 10.43.91.235 192.168.100.130 80:32055/TCP,443:31010/TCP 50s
service/ingress-nginx-controller-admission ClusterIP 10.43.14.197 <none> 443/TCP 50s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/ingress-nginx-controller 1/1 1 1 50s
NAME COMPLETIONS DURATION AGE
job.batch/ingress-nginx-admission-create 1/1 8s 50s
job.batch/ingress-nginx-admission-patch 1/1 8s 50s
NAME WEBHOOKS AGE
validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission 1 50s
ingress-nginx-controllerにEXTERNAL-IPが振られている事が確認できました。
yumenomatayumeこの状態でjupyter-notebookをデプロイしてみます。
$ git clone git@github.com:ymmmtym/manifests.git
$ cd manifests
$ kubectl apply -k jupyter-notebook
$ kubectl get ing
NAME CLASS HOSTS ADDRESS PORTS AGE
jupyter-notebook <none> jupyter-notebook.ymmmtym.com.local 192.168.100.4,192.168.100.5,192.168.100.6 80 61s
HOSTはjupyter-notebook.ymmmtym.com.local にしております。
ADDRESSはNodePortで表示されていますが、先ほど取得してingress-nginx-controllerにHTTPアクセスしてみます。
$ curl -I 192.168.100.130
HTTP/1.1 404 Not Found
Date: Sun, 25 Jul 2021 08:39:58 GMT
Content-Type: text/html
Content-Length: 146
Connection: keep-alive
$ curl -I 192.168.100.130 -H 'Host: jupyter-notebook.ymmmtym.com.local'
HTTP/1.1 405 Method Not Allowed
Date: Sun, 25 Jul 2021 08:40:02 GMT
Content-Type: text/html; charset=UTF-8
Content-Length: 87
Connection: keep-alive
名前解決はできていないので、IPアドレスを指定してアクセスしました。
以前までは-H 'Host: jupyter-notebook.ymmmtym.com.local'でアクセスしても404となってしまいましたが、今回は405となっておりアクセス出来ていそうです。
yumenomatayume暫定的に/etc/hostsを修正してアクセスできるようにしてみます。
$ sudo vim /etc/hosts
$ tail -1 /etc/hosts # 以下の1行を追記
192.168.100.130 jupyter-notebook.ymmmtym.com.local
$ open http://jupyter-notebook.ymmmtym.com.local
ブラウザで開いてみると、、、
アクセスする事ができました！！
次回は以下の設定をしていこうと思います。
- DNSで名前解決できるようにする
- HTTPS
このスクラップは2021/07/26にクローズされました