Zenn
Closed6

RKEで構築したオンプレKubernetesクラスタにIngress設定

Kubernetes
ingress
rke
MetalLB
オンプレ
yumenomatayumeyumenomatayume

以下のパッチを当ててtype: LoadBalancerに変更します。

kubectl patch svc default-http-backend -n ingress-nginx -p '{"spec": {"type": "LoadBalancer"}}'

EXTERNAL-IPにmetallbのセグメントのIPアドレスが割り振られていることを確認します。

$ kubectl get svc -n  ingress-nginx
NAME                   TYPE           CLUSTER-IP      EXTERNAL-IP       PORT(S)        AGE
default-http-backend   LoadBalancer   10.43.201.185   192.168.100.129   80:32270/TCP   14h

$ kubectl describe svc -n ingress-nginx
Name:                     default-http-backend
Namespace:                ingress-nginx
Labels:                   app=default-http-backend
Annotations:              <none>
Selector:                 app=default-http-backend
Type:                     LoadBalancer
IP Families:              <none>
IP:                       10.43.201.185
IPs:                      10.43.201.185
LoadBalancer Ingress:     192.168.100.129
Port:                     <unset>  80/TCP
TargetPort:               8080/TCP
NodePort:                 <unset>  32270/TCP
Endpoints:                10.42.0.4:8080
Session Affinity:         None
External Traffic Policy:  Cluster
Events:
  Type    Reason        Age                    From             Message
  ----    ------        ----                   ----             -------
  Normal  nodeAssigned  113s (x9482 over 14h)  metallb-speaker  announcing from node "192.168.100.4"

実際にHTTPアクセスしてみます。

$ curl 192.168.100.129
default backend - 404
yumenomatayumeyumenomatayume

https://www.ibm.com/docs/ja/cem?topic=troubleshooting-login-error-default-backend-404

上記によるとingress-controllerの設定を調べる必要があります。
RKEの場合は名前が変わっているようなので、以下で確認します。。

$ kubectl get configmap -n kube-system
NAME                                 DATA   AGE
coredns                              1      14h
coredns-autoscaler                   1      14h
extension-apiserver-authentication   6      14h
full-cluster-state                   1      14h
kube-flannel-cfg                     2      14h
kube-root-ca.crt                     1      14h
rke-coredns-addon                    1      14h
rke-ingress-controller               1      14h
rke-metrics-addon                    1      14h
rke-network-plugin                   1      14h

rke-ingress-controllerを見てみます。

$ kubectl get configmap -n kube-system rke-ingress-controller -o yaml
apiVersion: v1
data:
  rke-ingress-controller: "\n# this template is intended for use by >= nginx-0.32.0\napiVersion:
    v1\nkind: Namespace\nmetadata:\n  name: ingress-nginx\n---\nkind: ConfigMap\napiVersion:
    v1\nmetadata:\n  name: nginx-configuration\n  namespace: ingress-nginx\n  labels:\n
    \   app: ingress-nginx\ndata:\n\n---\nkind: ConfigMap\napiVersion: v1\nmetadata:\n
    \ name: tcp-services\n  namespace: ingress-nginx\n---\nkind: ConfigMap\napiVersion:
    v1\nmetadata:\n  name: udp-services\n  namespace: ingress-nginx\n\n---\napiVersion:
    v1\nkind: ServiceAccount\nmetadata:\n  name: nginx-ingress-serviceaccount\n  namespace:
    ingress-nginx\n---\napiVersion: rbac.authorization.k8s.io/v1beta1\nkind: ClusterRole\nmetadata:\n
    \ name: nginx-ingress-clusterrole\nrules:\n  - apiGroups:\n      - \"\"\n    resources:\n
    \     - configmaps\n      - endpoints\n      - nodes\n      - pods\n      - secrets\n
    \   verbs:\n      - list\n      - watch\n  - apiGroups:\n      - \"\"\n    resources:\n
    \     - nodes\n    verbs:\n      - get\n  - apiGroups:\n      - \"\"\n    resources:\n
    \     - services\n    verbs:\n      - get\n      - list\n      - update\n      -
    watch\n  - apiGroups:\n      - extensions\n      - \"networking.k8s.io\" # k8s
    1.14+\n    resources:\n      - ingresses\n    verbs:\n      - get\n      - list\n
    \     - watch\n  - apiGroups:\n      - \"\"\n    resources:\n      - events\n
    \   verbs:\n      - create\n      - patch\n  - apiGroups:\n      - extensions\n
    \     - \"networking.k8s.io\" # k8s 1.14+\n    resources:\n      - ingresses/status\n
    \   verbs:\n      - update\n---\napiVersion: rbac.authorization.k8s.io/v1beta1\nkind:
    Role\nmetadata:\n  name: nginx-ingress-role\n  namespace: ingress-nginx\nrules:\n
    \ - apiGroups:\n      - \"\"\n    resources:\n      - namespaces\n    verbs:\n
    \     - get\n  - apiGroups:\n      - \"\"\n    resources:\n      - configmaps\n
    \     - pods\n      - secrets\n      - endpoints\n    verbs:\n      - get\n      -
    list\n      - watch\n  - apiGroups:\n      - \"\"\n    resources:\n      - services\n
    \   verbs:\n      - get\n      - list\n      - update\n      - watch\n  - apiGroups:\n
    \     - extensions\n      - \"networking.k8s.io\" # k8s 1.14+\n    resources:\n
    \     - ingresses\n    verbs:\n      - get\n      - list\n      - watch\n  - apiGroups:\n
    \     - extensions\n      - \"networking.k8s.io\" # k8s 1.14+\n    resources:\n
    \     - ingresses/status\n    verbs:\n      - update\n  - apiGroups:\n      -
    \"\"\n    resources:\n      - configmaps\n    resourceNames:\n      # Defaults
    to \"<election-id>-<ingress-class>\"\n      # Here: \"<ingress-controller-leader>-<nginx>\"\n
    \     # This has to be adapted if you change either parameter\n      # when launching
    the nginx-ingress-controller.\n      - ingress-controller-leader-nginx\n    verbs:\n
    \     - get\n      - update\n  - apiGroups:\n      - \"\"\n    resources:\n      -
    configmaps\n    verbs:\n      - create\n  - apiGroups:\n      - \"\"\n    resources:\n
    \     - endpoints\n    verbs:\n      - create\n      - get\n      - update\n  -
    apiGroups:\n      - \"\"\n    resources:\n      - events\n    verbs:\n      -
    create\n      - patch\n---\napiVersion: rbac.authorization.k8s.io/v1beta1\nkind:
    RoleBinding\nmetadata:\n  name: nginx-ingress-role-nisa-binding\n  namespace:
    ingress-nginx\nroleRef:\n  apiGroup: rbac.authorization.k8s.io\n  kind: Role\n
    \ name: nginx-ingress-role\nsubjects:\n  - kind: ServiceAccount\n    name: nginx-ingress-serviceaccount\n
    \   namespace: ingress-nginx\n---\napiVersion: rbac.authorization.k8s.io/v1beta1\nkind:
    ClusterRoleBinding\nmetadata:\n  name: nginx-ingress-clusterrole-nisa-binding\nroleRef:\n
    \ apiGroup: rbac.authorization.k8s.io\n  kind: ClusterRole\n  name: nginx-ingress-clusterrole\nsubjects:\n
    \ - kind: ServiceAccount\n    name: nginx-ingress-serviceaccount\n    namespace:
    ingress-nginx\n\n---\napiVersion: apps/v1\nkind: DaemonSet\nmetadata:\n  name:
    nginx-ingress-controller\n  namespace: ingress-nginx\nspec:\n  selector:\n    matchLabels:\n
    \     app: ingress-nginx\n\n  updateStrategy:\n    rollingUpdate:\n      maxUnavailable:
    1\n    type: RollingUpdate\n\n  template:\n    metadata:\n      labels:\n        app:
    ingress-nginx\n      annotations:\n        prometheus.io/port: '10254'\n        prometheus.io/scrape:
    'true'\n    spec:\n      affinity:\n        nodeAffinity:\n          requiredDuringSchedulingIgnoredDuringExecution:\n
    \           nodeSelectorTerms:\n              - matchExpressions:\n                -
    key: beta.kubernetes.io/os\n                  operator: NotIn\n                  values:\n
    \                   - windows\n                - key: node-role.kubernetes.io/worker\n
    \                 operator: Exists\n      hostNetwork: true\n      \n# Rancher
    specific change\n\n      \n      serviceAccountName: nginx-ingress-serviceaccount\n
    \     \n      terminationGracePeriodSeconds: 60\n      tolerations:\n      - effect:
    NoExecute\n        operator: Exists\n      - effect: NoSchedule\n        operator:
    Exists\n      containers:\n        - name: nginx-ingress-controller\n          image:
    rancher/nginx-ingress-controller:nginx-0.43.0-rancher3\n          args:\n            -
    /nginx-ingress-controller\n            - --default-backend-service=$(POD_NAMESPACE)/default-http-backend\n
    \           - --configmap=$(POD_NAMESPACE)/nginx-configuration\n            -
    --election-id=ingress-controller-leader\n            - --ingress-class=nginx\n
    \           - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services\n            -
    --udp-services-configmap=$(POD_NAMESPACE)/udp-services\n            - --annotations-prefix=nginx.ingress.kubernetes.io\n
    \         \n          securityContext:\n            capabilities:\n                drop:\n
    \               - ALL\n                add:\n                - NET_BIND_SERVICE\n
    \           runAsUser: 101\n          env:\n            - name: POD_NAME\n              valueFrom:\n
    \               fieldRef:\n                  fieldPath: metadata.name\n            -
    name: POD_NAMESPACE\n              valueFrom:\n                fieldRef:\n                  fieldPath:
    metadata.namespace\n\n          ports:\n          - name: http\n            containerPort:
    80\n          - name: https\n            containerPort: 443\n          livenessProbe:\n
    \           failureThreshold: 3\n            httpGet:\n              path: /healthz\n
    \             port: 10254\n              scheme: HTTP\n            initialDelaySeconds:
    10\n            periodSeconds: 10\n            successThreshold: 1\n            timeoutSeconds:
    1\n          readinessProbe:\n            failureThreshold: 3\n            httpGet:\n
    \             path: /healthz\n              port: 10254\n              scheme:
    HTTP\n            initialDelaySeconds: 10\n            periodSeconds: 10\n            successThreshold:
    1\n            timeoutSeconds: 1\n\n\n---\napiVersion: apps/v1\nkind: Deployment\nmetadata:\n
    \ name: default-http-backend\n  labels:\n    app: default-http-backend\n  namespace:
    ingress-nginx\nspec:\n  replicas: 1\n  selector:\n    matchLabels:\n      app:
    default-http-backend\n  template:\n    metadata:\n      labels:\n        app:
    default-http-backend\n    spec:\n      affinity:\n        nodeAffinity:\n          requiredDuringSchedulingIgnoredDuringExecution:\n
    \           nodeSelectorTerms:\n              - matchExpressions:\n                -
    key: beta.kubernetes.io/os\n                  operator: NotIn\n                  values:\n
    \                   - windows\n                - key: node-role.kubernetes.io/worker\n
    \                 operator: Exists\n      terminationGracePeriodSeconds: 60\n#
    Rancher specific change\n      tolerations:\n      - effect: NoExecute\n        operator:
    Exists\n      - effect: NoSchedule\n        operator: Exists\n      containers:\n
    \     - name: default-http-backend\n        # Any image is permissable as long
    as:\n        # 1. It serves a 404 page at /\n        # 2. It serves 200 on a /healthz
    endpoint\n        image: rancher/mirrored-nginx-ingress-controller-defaultbackend:1.5-rancher1\n
    \       livenessProbe:\n          httpGet:\n            path: /healthz\n            port:
    8080\n            scheme: HTTP\n          initialDelaySeconds: 30\n          timeoutSeconds:
    5\n        ports:\n        - containerPort: 8080\n        resources:\n          limits:\n
    \           cpu: 10m\n            memory: 20Mi\n          requests:\n            cpu:
    10m\n            memory: 20Mi\n---\napiVersion: v1\nkind: Service\nmetadata:\n
    \ name: default-http-backend\n  namespace: ingress-nginx\n  labels:\n    app:
    default-http-backend\nspec:\n  ports:\n  - port: 80\n    targetPort: 8080\n  selector:\n
    \   app: default-http-backend\n"
kind: ConfigMap
metadata:
  creationTimestamp: "2021-07-24T16:08:08Z"
  name: rke-ingress-controller
  namespace: kube-system
  resourceVersion: "583"
  uid: 722e18f7-133f-49f4-a36f-209ddb196c92

そもそもhttp2: falseという項目はなさそうでした。

yumenomatayumeyumenomatayume

https://www.youtube.com/watch?v=iqVt5mbvlJ0
https://gitlab.com/monachus/channel/-/tree/master/resources/2020.01

上記にRKEでのセットアップ方法が公開されていたので、試してみます。
metallbはデプロイ完了しているので、nginxからデプロイしてみます。

git clone https://gitlab.com/monachus/channel.git
cd resources/2020.01/deploy/2-nginx

applyしてみます。

$ kubectl apply -k .
namespace/ingress-nginx configured
serviceaccount/ingress-nginx created
serviceaccount/ingress-nginx-admission created
role.rbac.authorization.k8s.io/ingress-nginx created
role.rbac.authorization.k8s.io/ingress-nginx-admission created
clusterrole.rbac.authorization.k8s.io/ingress-nginx created
clusterrole.rbac.authorization.k8s.io/ingress-nginx-admission created
rolebinding.rbac.authorization.k8s.io/ingress-nginx created
rolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
configmap/ingress-nginx-controller created
configmap/tcp-services configured
configmap/udp-services configured
service/ingress-nginx-controller created
service/ingress-nginx-controller-admission created
deployment.apps/ingress-nginx-controller created
job.batch/ingress-nginx-admission-create created
job.batch/ingress-nginx-admission-patch created
validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission created

作成されたリソースは以下になっています。

$ kubectl get -k .
NAME                      STATUS   AGE
namespace/ingress-nginx   Active   16h

NAME                                     SECRETS   AGE
serviceaccount/ingress-nginx             1         50s
serviceaccount/ingress-nginx-admission   1         50s

NAME                                                     CREATED AT
role.rbac.authorization.k8s.io/ingress-nginx             2021-07-25T08:20:34Z
role.rbac.authorization.k8s.io/ingress-nginx-admission   2021-07-25T08:20:34Z

NAME                                                            CREATED AT
clusterrole.rbac.authorization.k8s.io/ingress-nginx             2021-07-25T08:20:34Z
clusterrole.rbac.authorization.k8s.io/ingress-nginx-admission   2021-07-25T08:20:34Z

NAME                                                            ROLE                           AGE
rolebinding.rbac.authorization.k8s.io/ingress-nginx             Role/ingress-nginx             50s
rolebinding.rbac.authorization.k8s.io/ingress-nginx-admission   Role/ingress-nginx-admission   50s

NAME                                                                   ROLE                                  AGE
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx             ClusterRole/ingress-nginx             50s
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx-admission   ClusterRole/ingress-nginx-admission   50s

NAME                                 DATA   AGE
configmap/ingress-nginx-controller   0      50s
configmap/tcp-services               0      16h
configmap/udp-services               0      16h

NAME                                         TYPE           CLUSTER-IP     EXTERNAL-IP       PORT(S)                      AGE
service/ingress-nginx-controller             LoadBalancer   10.43.91.235   192.168.100.130   80:32055/TCP,443:31010/TCP   50s
service/ingress-nginx-controller-admission   ClusterIP      10.43.14.197   <none>            443/TCP                      50s

NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/ingress-nginx-controller   1/1     1            1           50s

NAME                                       COMPLETIONS   DURATION   AGE
job.batch/ingress-nginx-admission-create   1/1           8s         50s
job.batch/ingress-nginx-admission-patch    1/1           8s         50s

NAME                                                                                  WEBHOOKS   AGE
validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission   1          50s

ingress-nginx-controllerにEXTERNAL-IPが振られている事が確認できました。

yumenomatayumeyumenomatayume

この状態でjupyter-notebookをデプロイしてみます。

https://github.com/ymmmtym/manifests/tree/main/jupyter-notebook

$ git clone git@github.com:ymmmtym/manifests.git
$ cd manifests
$ kubectl apply -k jupyter-notebook

$ kubectl get ing
NAME               CLASS    HOSTS                                ADDRESS                                     PORTS   AGE
jupyter-notebook   <none>   jupyter-notebook.ymmmtym.com.local   192.168.100.4,192.168.100.5,192.168.100.6   80      61s

HOSTはjupyter-notebook.ymmmtym.com.local にしております。
ADDRESSはNodePortで表示されていますが、先ほど取得してingress-nginx-controllerにHTTPアクセスしてみます。

$ curl -I 192.168.100.130
HTTP/1.1 404 Not Found
Date: Sun, 25 Jul 2021 08:39:58 GMT
Content-Type: text/html
Content-Length: 146
Connection: keep-alive

$ curl -I 192.168.100.130 -H 'Host: jupyter-notebook.ymmmtym.com.local'
HTTP/1.1 405 Method Not Allowed
Date: Sun, 25 Jul 2021 08:40:02 GMT
Content-Type: text/html; charset=UTF-8
Content-Length: 87
Connection: keep-alive

名前解決はできていないので、IPアドレスを指定してアクセスしました。
以前までは-H 'Host: jupyter-notebook.ymmmtym.com.local'でアクセスしても404となってしまいましたが、今回は405となっておりアクセス出来ていそうです。

yumenomatayumeyumenomatayume

暫定的に/etc/hostsを修正してアクセスできるようにしてみます。

$ sudo vim /etc/hosts
$ tail -1 /etc/hosts # 以下の1行を追記
192.168.100.130 jupyter-notebook.ymmmtym.com.local

$ open http://jupyter-notebook.ymmmtym.com.local

ブラウザで開いてみると、、、

Imgur

アクセスする事ができました！！

次回は以下の設定をしていこうと思います。

  • DNSで名前解決できるようにする
  • HTTPS
このスクラップは2021/07/26にクローズされました