🔎

MeiliSearch使ってみる kubernetes編

49 min read

使い方はいくらかわかってきたので、運用面を考えてみます

前回までの話

  1. MeiliSearchを使ってみる
  2. MeiliSearch使ってみる Rails編
  3. MeiliSearch使ってみる Rust編

運用

  • k8s内で使いたい
    • helm chartがあるのでhelmでデプロイ&バージョン管理する
  • 冗長化したい
    • 冗長化した場合永続化されたデータはどうやってどこに持つのだろうか?
    • そもそもできるのか

環境

準備

minikube, kubectlのインストール

https://qiita.com/yuichi1992_west/items/571016084c110d15320e

参考にさせていただきました 🙏

minikubeの起動

過去の設定が悪さしているっぽくエラーで起動できなかったので一回削除しました

❯ minikube start  --kubernetes-version=v1.22.2
😄  Ubuntu 20.04 上の minikube v1.23.2
✨  dockerドライバーが自動的に選択されました。他の選択肢:  virtualbox, none, ssh
👍  コントロールプレーンのノード minikube を minikube 上で起動しています
🚜  イメージを Pull しています...
🔥  docker container (CPUs=2, Memory=16000MB) を作成しています...
🐳  Docker 20.10.8 で Kubernetes v1.22.2 を準備しています...
    ▪ 証明書と鍵を作成しています...
    ▪ Control Plane を起動しています...
    ▪ RBAC のルールを設定中です.../ E1009 01:11:25.835511   66512 start.go:142] Unable to scale down deployment "coredns" in namespace "kube-system" to 1 replica: timed out waiting for the condition

🔎  Kubernetes コンポーネントを検証しています...
    ▪ イメージ gcr.io/k8s-minikube/storage-provisioner:v5 を使用しています
❗  'default-storageclass' を有効にする際にエラーが発生しました。running callbacks: [Error making standard the default storage class: Error listing StorageClasses: Unauthorized]
🌟  有効なアドオン: storage-provisioner


❌  Exiting due to K8S_UNHEALTHY_CONTROL_PLANE: wait 6m0s for node: wait for healthy API server: controlPlane never updated to v1.22.2
💡  提案: Control Plane could not update, try minikube delete --all --purge
🍿  Related issue: https://github.com/kubernetes/minikube/issues/11417

╭───────────────────────────────────────────────────────────────────────────────────────────╮
│                                                                                           │
│    😿  If the above advice does not help, please let us know:                             │
│    👉  https://github.com/kubernetes/minikube/issues/new/choose                           │
│                                                                                           │
│    Please run `minikube logs --file=logs.txt` and attach logs.txt to the GitHub issue.    │
│                                                                                           │
╰───────────────────────────────────────────────────────────────────────────────────────────╯
❯ minikube delete --all --purge
🔥  docker の「minikube」を削除しています...
🔥  /home/xxxx/.minikube/machines/minikube を削除しています...
💀  クラスタ "minikube" の全てのトレースを削除しました。
🔥  全てのプロファイルの削除に成功しました
💀  Successfully purged minikube directory located at - [/home/xxxx/.minikube]
❯ minikube start  --kubernetes-version=v1.22.2
😄  Ubuntu 20.04 上の minikube v1.23.2
✨  dockerドライバーが自動的に選択されました。他の選択肢:  virtualbox, none, ssh
👍  コントロールプレーンのノード minikube を minikube 上で起動しています
🚜  イメージを Pull しています...
💾  Kubernetes v1.22.2 のダウンロードの準備をしています
    > preloaded-images-k8s-v13-v1...: 511.84 MiB / 511.84 MiB  100.00% 76.70 Mi
🔥  docker container (CPUs=2, Memory=16000MB) を作成しています...
🐳  Docker 20.10.8 で Kubernetes v1.22.2 を準備しています...
    ▪ 証明書と鍵を作成しています...
    ▪ Control Plane を起動しています...
    ▪ RBAC のルールを設定中です...
🔎  Kubernetes コンポーネントを検証しています...
    ▪ イメージ gcr.io/k8s-minikube/storage-provisioner:v5 を使用しています
🌟  有効なアドオン: default-storageclass, storage-provisioner
🏄  完了しました! kubectl が「"minikube"」クラスタと「"default"」ネームスペースを使用するよう構成されました
❯ minikube status
minikube
type: Control Plane
host: Running
kubelet: Running
apiserver: Running
kubeconfig: Configured

❯ kubectl get po
No resources found in default namespace.

大丈夫そうです。

Helmのインストール

https://helm.sh/ja/docs/intro/install/

MeiliSearchのインストール

https://github.com/meilisearch/meilisearch-kubernetes/blob/main/README.md#deploy-meilisearch-using-helm-
READMEを見て進めます。

Helm repo

❯ helm repo add meilisearch https://meilisearch.github.io/meilisearch-kubernetes
"meilisearch" has been added to your repositories

install

❯ helm upgrade -i meilisearch meilisearch/meilisearch
Release "meilisearch" does not exist. Installing it now.
NAME: meilisearch
LAST DEPLOYED: Sat Oct  9 01:25:08 2021
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTES:
1. Get the application URL by running these commands:
  kubectl port-forward --namespace default svc/meilisearch 7700:7700 &
  echo "Visit http://127.0.0.1:7700 to use your application"
❯ helm list
NAME            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                   APP VERSION
meilisearch     default         1               2021-10-09 01:25:08.954581332 +0900 JST deployed        meilisearch-0.1.17      v0.22.0
❯ kubectl get po
NAME            READY   STATUS    RESTARTS   AGE
meilisearch-0   0/1     Running   0          53s

起動したっぽいですね

アクセスしてみる

kubectl port-forward でアクセスしてみます

❯ kubectl port-forward svc/meilisearch 7700:7700
Forwarding from 127.0.0.1:7700 -> 7700
Forwarding from [::1]:7700 -> 7700
Handling connection for 7700
curl -v http://localhost:7700
*   Trying 127.0.0.1:7700...
* Connected to localhost (127.0.0.1) port 7700 (#0)
> GET / HTTP/1.1
> Host: localhost:7700
> User-Agent: curl/7.73.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-length: 2191
< content-type: text/html
< date: Fri, 08 Oct 2021 16:31:39 GMT
<
<!doctype html><html lang="en"><head><meta charset="utf-8"/><link rel="icon" href="/favicon.ico"/><meta name="viewport" content="width=device-width,initial-scale=1"/><meta name="theme-color" content="#000000"/><meta name="description" content="Dashboard to test MeiliSearch's search engine"/><link rel="apple-touch-icon" href="/logo.png"/><link rel="manifest" href="/manifest.json"/><title>Mini-dashboard | MeiliSearch</title></head><body><noscript>You need to enable JavaScript to run this app.</noscript><div id="root"></div><script>!function(e){function r(r){for(var n,i,a=r[0],l=r[1],f=r[2],c=0,s=[];c<a.length;c++)i=a[c],Object.prototype.hasOwnProperty.call(o,i)&&o[i]&&s.push(o[i][0]),o[i]=0;for(n in l)Object.prototype.hasOwnProperty.call(l,n)&&(e[n]=l[n]);for(p&&p(r);s.length;)s.shift()();return u.push.apply(u,f||[]),t()}function t(){for(var e,r=0;r<u.length;r++){for(var t=u[r],n=!0,a=1;a<t.length;a++){var l=t[a];0!==o[l]&&(n=!1)}n&&(u.splice(r--,1),e=i(i.s=t[0]))}return e}var n={},o={1:0},u=[];function i(r){if(n[r])return n[r].exports;var t=n[r]={i:r,l:!1,exports:{}};return e[r].call(t.exports,t,t.exports,i),t.l=!0,t.exports}i.m=e,i.c=n,i.d=function(e,r,t){i.o(e,r)||Object.defineProperty(e,r,{enumerable:!0,get:t})},i.r=function(e){"undefined"!=typeof Symbol&&Symbol.toStringTag&&Object.defineProperty(e,Symbol.toStringTag,{value:"Module"}),Object.defineProperty(e,"__esModule",{value:!0})},i.t=function(e,r){if(1&r&&(e=i(e)),8&r)return e;if(4&r&&"object"==typeof e&&e&&e.__esModule)return e;var t=Object.create(null);if(i.r(t),Object.defineProperty(t,"default",{enumerable:!0,value:e}),2&r&&"string"!=typeof e)for(var n in e)i.d(t,n,function(r){return e[r]}.bind(null,n));return t},i.n=function(e){var r=e&&e.__esModule?function(){return e.default}:function(){return e};return i.d(r,"a",r),r},i.o=function(e,r){return Object.prototype.hasOwnProperty.call(e,r)},i.p="/";var a=this["webpackJsonpmini-dashboard"]=this["webpackJsonpmini-dashboard"]||[],l=a.push.bind(a);a.push=r,a=a.slice();for(var f=0;f<a.length;f++)r(a[f]);var p=l* Connection #0 to host localhost left intact
;t()}([])</script><script src="/static/js/2.b22bb78b.chunk.js"></script><script src="/static/js/main.81816167.chunk.js"></script></body></html>%               
curl -v http://localhost:7700/indexes
*   Trying 127.0.0.1:7700...
* Connected to localhost (127.0.0.1) port 7700 (#0)
> GET /indexes HTTP/1.1
> Host: localhost:7700
> User-Agent: curl/7.73.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-length: 2
< content-type: application/json
< date: Fri, 08 Oct 2021 16:34:10 GMT
<
* Connection #0 to host localhost left intact
[]%                                                                                                               

アクセスできました

冗長化したい

helm chartのテンプレートを確認する

まずは、helm chartの中身を確認してどういう構成になっているのか見てみます

❯ helm template meilisearch/meilisearch
---
# Source: meilisearch/templates/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: RELEASE-NAME-meilisearch
  labels:
    app.kubernetes.io/name: meilisearch
    helm.sh/chart: meilisearch-0.1.17
    app.kubernetes.io/instance: RELEASE-NAME
    app.kubernetes.io/managed-by: Helm
---
# Source: meilisearch/templates/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: RELEASE-NAME-meilisearch-environment
  labels:
    app.kubernetes.io/name: meilisearch
    helm.sh/chart: meilisearch-0.1.17
    app.kubernetes.io/instance: RELEASE-NAME
    app.kubernetes.io/managed-by: Helm
data:
  MEILI_ENV: "development"
  MEILI_NO_ANALYTICS: "true"
---
# Source: meilisearch/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: RELEASE-NAME-meilisearch
  labels:
    app.kubernetes.io/name: meilisearch
    helm.sh/chart: meilisearch-0.1.17
    app.kubernetes.io/instance: RELEASE-NAME
    app.kubernetes.io/managed-by: Helm
spec:
  type: ClusterIP
  ports:
    - port: 7700
      targetPort: http
      protocol: TCP
      name: http
  selector:
    app.kubernetes.io/name: meilisearch
    app.kubernetes.io/instance: RELEASE-NAME
---
# Source: meilisearch/templates/statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: RELEASE-NAME-meilisearch
  labels:
    app.kubernetes.io/name: meilisearch
    helm.sh/chart: meilisearch-0.1.17
    app.kubernetes.io/instance: RELEASE-NAME
    app.kubernetes.io/managed-by: Helm
spec:
  replicas: 1
  serviceName: RELEASE-NAME-meilisearch
  selector:
    matchLabels:
      app.kubernetes.io/name: meilisearch
      app.kubernetes.io/instance: RELEASE-NAME
  template:
    metadata:
      labels:
        app.kubernetes.io/name: meilisearch
        app.kubernetes.io/instance: RELEASE-NAME
    spec:
      serviceAccountName: RELEASE-NAME-meilisearch
      containers:
        - name: meilisearch
          image: "getmeili/meilisearch:v0.22.0"
          imagePullPolicy: IfNotPresent
          envFrom:
          - configMapRef:
              name: RELEASE-NAME-meilisearch-environment
          ports:
            - name: http
              containerPort: 7700
              protocol: TCP
          livenessProbe:
            httpGet:
              path: /health
              port: http
            periodSeconds: 60
            initialDelaySeconds: 60
          readinessProbe:
            httpGet:
              path: /health
              port: http
            periodSeconds: 60
            initialDelaySeconds: 60
          resources:
            {}
---
# Source: meilisearch/templates/tests/test-connection.yaml
apiVersion: v1
kind: Pod
metadata:
  name: release-name-meilisearch-test-connection
  labels:
    app.kubernetes.io/name: meilisearch
    helm.sh/chart: meilisearch-0.1.17
    app.kubernetes.io/instance: RELEASE-NAME
    app.kubernetes.io/managed-by: Helm
  annotations:
    "helm.sh/hook": test-success
spec:
  containers:
    - name: wget
      image: busybox
      command: ['wget']
      args:  ['RELEASE-NAME-meilisearch:7700']
  restartPolicy: Never

なるほど StatefulSet で起動されているんですね。

https://github.com/meilisearch/meilisearch-kubernetes/blob/b5faa95143df897bd786105b31c1e20dcce83022/charts/meilisearch/templates/statefulset.yaml#L38-L43

この辺を見ると、persistence.enabledをtrueにすればpvcを作ってデータを保存できそうな感じですかね

https://github.com/meilisearch/meilisearch-kubernetes/blob/b5faa95143df897bd786105b31c1e20dcce83022/charts/meilisearch/templates/statefulset.yaml#L14

オートスケールはできなそうですが、↑ここでreplicasを指定できるので常時2台以上起動するようなことはできそうですかね。
自前でHPAを書けばオートスケールできるだろうか? StatefulSetにHPAを設定したことがないので後で確認する

kubectlからhelmでデプロイされたリソースを確認してみます。

❯ kubectl get po
NAME            READY   STATUS    RESTARTS     AGE
meilisearch-0   1/1     Running   1 (3d ago)   3d20h

❯ kubectl get svc
NAME          TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
kubernetes    ClusterIP   10.96.0.1      <none>        443/TCP    3d20h
meilisearch   ClusterIP   10.107.240.7   <none>        7700/TCP   3d20h

❯ kubectl get statefulset
NAME          READY   AGE
meilisearch   1/1     3d20h

❯ kubectl get hpa
No resources found in default namespace.

❯ kubectl get cm
NAME                      DATA   AGE
kube-root-ca.crt          1      3d20h
meilisearch-environment   2      3d20h

❯ kubectl get sa
NAME          SECRETS   AGE
default       1         3d20h
meilisearch   1         3d20h

helm chartのvaluesを確認する

なにを拡張できるのか、helmのvaluesを見てみます。

❯ helm show values meilisearch/meilisearch
# Default values for <CHARTNAME>.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.

replicaCount: 1

readinessProbe:
  periodSeconds: 60
  InitialDelaySeconds: 60

livenessProbe:
  periodSeconds: 60
  InitialDelaySeconds: 60

image:
  repository: getmeili/meilisearch
  tag: v0.22.0
  pullPolicy: IfNotPresent

nameOverride: ""
fullnameOverride: ""

# Environment loaded into the configMap
environment:
  MEILI_NO_ANALYTICS: true
  MEILI_ENV: development
  # For production deployment, the environment MEILI_MASTER_KEY is required.
  # If MEILI_ENV is set to "production" without setting MEILI_MASTER_KEY, this
  # chart will automatically create a secure MEILI_MASTER_KEY and push it as a
  # secret. Otherwise the below value of MEILI_MASTER_KEY will be used instead.
  # MEILI_MASTER_KEY:

podAnnotations: {}

customLabels: {}

service:
  type: ClusterIP
  port: 7700

container:
  containerPort: 7700

ingress:
  enabled: false
  annotations: {}
    # kubernetes.io/ingress.class: nginx
  path: /
  hosts:
    - meilisearch-example.local
  tls: []
  #  - secretName: chart-example-tls
  #    hosts:
  #      - chart-example.local

persistence:
  enabled: false
  accessMode: ReadWriteOnce
  ## Persistent Volume Storage Class
  ## If defined, storageClassName: <storageClass>
  ## If set to "-", storageClassName: "", which disables dynamic provisioning
  ## If undefined (the default) or set to null, no storageClassName spec is
  ##   set, choosing the default provisioner.  (gp2 on AWS, standard on
  ##   GKE, AWS & OpenStack)
  ##
  # storageClass: "-"
  size: 10Gi
  volume:
    name: data
    mountPath: /data.ms

resources: {}
  # We usually recommend not to specify default resources and to leave this as a conscious
  # choice for the user. This also increases chances charts run on environments with little
  # resources, such as Minikube. If you do want to specify resources, uncomment the following
  # lines, adjust them as necessary, and remove the curly braces after 'resources:'.
  # limits:
  #  cpu: 100m
  #  memory: 128Mi
  # requests:
  #  cpu: 100m
  #  memory: 128Mi

nodeSelector: {}

tolerations: []

affinity: {}

valuesを独自に設定してみる(PVCなし)

まず事前に多少データを登録しておきます。

curl \
  -H "Content-Type: application/json" \
  -X POST 'http://localhost:7700/indexes/movies/documents' \
  --data '[{
      "id": 287947,
      "title": "Shazam",
      "poster": "https://image.tmdb.org/t/p/w1280/xnopI5Xtky18MPhK40cZAGAOVeV.jpg",
      "overview": "A boy is given the ability to become an adult superhero in times of need with a single magic word.",
      "release_date": "2019-03-23"
  }]'
{"updateId":0}%                                                                                                   ```

```shell
❯ curl -v http://localhost:7700/indexes/movies/documents
*   Trying 127.0.0.1:7700...
* Connected to localhost (127.0.0.1) port 7700 (#0)
> GET /indexes/movies/documents HTTP/1.1
> Host: localhost:7700
> User-Agent: curl/7.73.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-length: 248
< content-type: application/json
< date: Tue, 12 Oct 2021 13:34:36 GMT
<
* Connection #0 to host localhost left intact
[{"id":287947,"title":"Shazam","poster":"https://image.tmdb.org/t/p/w1280/xnopI5Xtky18MPhK40cZAGAOVeV.jpg","overview":"A boy is given the ability to become an adult superhero in times of need with a single magic word.","release_date":"2019-03-23"}]%                                                                    

とれました。

ではvaluesを設定して、2台起動してみます。

values.yaml
replicaCount: 2
upgrade.sh
#!/bin/bash -e

helm repo add meilisearch https://meilisearch.github.io/meilisearch-kubernetes
helm repo update
helm upgrade -i meilisearch meilisearch/meilisearch --version 0.1.18 -f values.yaml
❯ kubectl get po
NAME            READY   STATUS    RESTARTS   AGE
meilisearch-0   1/1     Running   0          10m
meilisearch-1   1/1     Running   0          99s

2台起動しました。
アクセスしてみます。

❯ kubectl port-forward svc/meilisearch 7700:7700
watch -n 0.5 "curl -v http://localhost:7700/indexes/movies/documents"
...

meilisearch-0 のログ

❯ kubectl logs -f meilisearch-0

888b     d888          d8b 888 d8b  .d8888b.                                    888
8888b   d8888          Y8P 888 Y8P d88P  Y88b                                   888
88888b.d88888              888     Y88b.                                        888
888Y88888P888  .d88b.  888 888 888  "Y888b.    .d88b.   8888b.  888d888 .d8888b 88888b.
888 Y888P 888 d8P  Y8b 888 888 888     "Y88b. d8P  Y8b     "88b 888P"  d88P"    888 "88b
888  Y8P  888 88888888 888 888 888       "888 88888888 .d888888 888    888      888  888
888   "   888 Y8b.     888 888 888 Y88b  d88P Y8b.     888  888 888    Y88b.    888  888
888       888  "Y8888  888 888 888  "Y8888P"   "Y8888  "Y888888 888     "Y8888P 888  888

Database path:          "./data.ms"
Server listening on:    "http://0.0.0.0:7700"
Environment:            "development"
Commit SHA:             "3172c96042747e1ec3d27666b2bbc5ca507c56a7"
Commit date:            "2021-10-11T14:27:46+00:00"
Package version:        "0.23.0"
Anonymous telemetry:    "Disabled"

No master key found; The server will accept unidentified requests. If you need some protection in development mode, please export a key: export MEILI_MASTER_KEY=xxx

Documentation:          https://docs.meilisearch.com
Source code:            https://github.com/meilisearch/meilisearch
Contact:                https://docs.meilisearch.com/resources/contact.html or bonjour@meilisearch.com

[2021-10-12T13:28:22Z INFO  actix_server::builder] Starting 24 workers
[2021-10-12T13:28:22Z INFO  actix_server::builder] Starting "actix-web-service-0.0.0.0:7700" service on 0.0.0.0:7700
[2021-10-12T13:29:28Z INFO  actix_web::middleware::logger] 127.0.0.1:33672 "GET / HTTP/1.1" 200 2191 "-" "curl/7.73.0" 0.000060
[2021-10-12T13:29:32Z INFO  actix_web::middleware::logger] 172.17.0.1:45660 "GET /health HTTP/1.1" 200 22 "-" "kube-probe/1.22" 0.000052
...
[2021-10-12T13:39:20Z INFO  actix_web::middleware::logger] 127.0.0.1:37512 "GET /indexes/movies/documents HTTP/1.1" 200 248 "-" "curl/7.73.0" 0.000330
[2021-10-12T13:39:20Z INFO  actix_web::middleware::logger] 127.0.0.1:37516 "GET /indexes/movies/documents HTTP/1.1" 200 248 "-" "curl/7.73.0" 0.000311
[2021-10-12T13:39:21Z INFO  actix_web::middleware::logger] 127.0.0.1:37524 "GET /indexes/movies/documents HTTP/1.1" 200 248 "-" "curl/7.73.0" 0.000412
[2021-10-12T13:39:21Z INFO  actix_web::middleware::logger] 127.0.0.1:37528 "GET /indexes/movies/documents HTTP/1.1" 200 248 "-" "curl/7.73.0" 0.000318
[2021-10-12T13:39:22Z INFO  actix_web::middleware::logger] 127.0.0.1:37536 "GET /indexes/movies/documents HTTP/1.1" 200 248 "-" "curl/7.73.0" 0.000357

meilisearch-1 のログ

❯ kubectl logs -f meilisearch-1

888b     d888          d8b 888 d8b  .d8888b.                                    888
8888b   d8888          Y8P 888 Y8P d88P  Y88b                                   888
88888b.d88888              888     Y88b.                                        888
888Y88888P888  .d88b.  888 888 888  "Y888b.    .d88b.   8888b.  888d888 .d8888b 88888b.
888 Y888P 888 d8P  Y8b 888 888 888     "Y88b. d8P  Y8b     "88b 888P"  d88P"    888 "88b
888  Y8P  888 88888888 888 888 888       "888 88888888 .d888888 888    888      888  888
888   "   888 Y8b.     888 888 888 Y88b  d88P Y8b.     888  888 888    Y88b.    888  888
888       888  "Y8888  888 888 888  "Y8888P"   "Y8888  "Y888888 888     "Y8888P 888  888

Database path:          "./data.ms"
Server listening on:    "http://0.0.0.0:7700"
Environment:            "development"
Commit SHA:             "3172c96042747e1ec3d27666b2bbc5ca507c56a7"
Commit date:            "2021-10-11T14:27:46+00:00"
Package version:        "0.23.0"
Anonymous telemetry:    "Disabled"

No master key found; The server will accept unidentified requests. If you need some protection in development mode, please export a key: export MEILI_MASTER_KEY=xxx

Documentation:          https://docs.meilisearch.com
Source code:            https://github.com/meilisearch/meilisearch
Contact:                https://docs.meilisearch.com/resources/contact.html or bonjour@meilisearch.com

[2021-10-12T13:37:22Z INFO  actix_server::builder] Starting 24 workers
[2021-10-12T13:37:23Z INFO  actix_server::builder] Starting "actix-web-service-0.0.0.0:7700" service on 0.0.0.0:7700
[2021-10-12T13:38:22Z INFO  actix_web::middleware::logger] 172.17.0.1:38196 "GET /health HTTP/1.1" 200 22 "-" "kube-probe/1.22" 0.000072
[2021-10-12T13:38:22Z INFO  actix_web::middleware::logger] 172.17.0.1:38198 "GET /health HTTP/1.1" 200 22 "-" "kube-probe/1.22" 0.000075
[2021-10-12T13:39:22Z INFO  actix_web::middleware::logger] 172.17.0.1:38908 "GET /health HTTP/1.1" 200 22 "-" "kube-probe/1.22" 0.000054
[2021-10-12T13:39:22Z INFO  actix_web::middleware::logger] 172.17.0.1:38906 "GET /health HTTP/1.1" 200 22 "-" "kube-probe/1.22" 0.000051
[2021-10-12T13:40:22Z INFO  actix_web::middleware::logger] 172.17.0.1:39666 "GET /health HTTP/1.1" 200 22 "-" "kube-probe/1.22" 0.000054
[2021-10-12T13:40:22Z INFO  actix_web::middleware::logger] 172.17.0.1:39668 "GET /health HTTP/1.1" 200 22 "-" "kube-probe/1.22" 0.000044
...

meilisearch-0 にしかアクセスが無いですね..

https://gomiba.co/archives/2019/01/2589/

サービスに対してフォワードしているわけではないので、フォワードしたポートに何度リクエストを送ってもロードバランスされない。

なるほど、そうなんですね。。
あまりローカルでk8s使わないので、これは知りませんでした。

脱線: minikubeで起動したサービスにアクセスしたい

https://kubernetes.io/ja/docs/setup/learning-environment/minikube/#サービス

ノードポート経由で公開されているサービスにアクセスするには、Minikubeを起動してアドレスを取得した後、シェルでこのコマンドを実行してください:
minikube service [-n NAMESPACE] [--url] NAME

なるほど?

❯ minikube service meilisearch
|-----------|-------------|-------------|--------------|
| NAMESPACE |    NAME     | TARGET PORT |     URL      |
|-----------|-------------|-------------|--------------|
| default   | meilisearch |             | No node port |
|-----------|-------------|-------------|--------------|
😿  サービス default/meilisearch は NodePort を持っていません

なるほど?

❯ kubectl describe svc/meilisearch
Name:              meilisearch
Namespace:         default
Labels:            app.kubernetes.io/instance=meilisearch
                   app.kubernetes.io/managed-by=Helm
                   app.kubernetes.io/name=meilisearch
                   helm.sh/chart=meilisearch-0.1.18
Annotations:       meta.helm.sh/release-name: meilisearch
                   meta.helm.sh/release-namespace: default
Selector:          app.kubernetes.io/instance=meilisearch,app.kubernetes.io/name=meilisearch
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.107.240.7
IPs:               10.107.240.7
Port:              http  7700/TCP
TargetPort:        http/TCP
Endpoints:         172.17.0.3:7700,172.17.0.4:7700
Session Affinity:  None
Events:            <none>

たしかに、ClusterIPになってますね。

https://github.com/meilisearch/meilisearch-kubernetes/blob/b5faa95143df897bd786105b31c1e20dcce83022/charts/meilisearch/templates/service.yaml#L14

service.type で変更できそうです。

values.yaml
replicaCount: 2
service:
  type: NodePort
❯ minikube service meilisearch
|-----------|-------------|-------------|---------------------------|
| NAMESPACE |    NAME     | TARGET PORT |            URL            |
|-----------|-------------|-------------|---------------------------|
| default   | meilisearch | http/7700   | http://192.168.49.2:31317 |
|-----------|-------------|-------------|---------------------------|
🎉  Opening service default/meilisearch in default browser...

きました

curl -v http://192.168.49.2:31317/indexes/movies/documents
*   Trying 192.168.49.2:31317...
* Connected to 192.168.49.2 (192.168.49.2) port 31317 (#0)
> GET /indexes/movies/documents HTTP/1.1
> Host: 192.168.49.2:31317
> User-Agent: curl/7.73.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 Not Found
< content-length: 168
< content-type: application/json
< date: Tue, 12 Oct 2021 13:59:51 GMT
<
* Connection #0 to host 192.168.49.2 left intact
{"message":"Index movies not found","errorCode":"index_not_found","errorType":"invalid_request_error","errorLink":"https://docs.meilisearch.com/errors#index_not_found"}%                                                                                                                                                   
❯ curl -v http://192.168.49.2:31317/indexes/movies/documents
*   Trying 192.168.49.2:31317...
* Connected to 192.168.49.2 (192.168.49.2) port 31317 (#0)
> GET /indexes/movies/documents HTTP/1.1
> Host: 192.168.49.2:31317
> User-Agent: curl/7.73.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-length: 248
< content-type: application/json
< date: Tue, 12 Oct 2021 13:59:53 GMT
<
* Connection #0 to host 192.168.49.2 left intact
[{"id":287947,"title":"Shazam","poster":"https://image.tmdb.org/t/p/w1280/xnopI5Xtky18MPhK40cZAGAOVeV.jpg","overview":"A boy is given the ability to become an adult superhero in times of need with a single magic word.","release_date":"2019-03-23"}]%                                                                   

うん

  • meilisearch-0 に行った時はデータが取れて 200
  • meilisearch-1 に行った時はindexがないので 404

になってますね。
OK

再起動してみる

再起動するとデータは消えるのか?
PVCが無いので消えそうな気がする

❯ kubectl rollout restart statefulset meilisearch
❯ kubectl get po
NAME            READY   STATUS    RESTARTS   AGE
meilisearch-0   1/1     Running   0          6m11s
meilisearch-1   1/1     Running   0          7m35s

再起動したのでアクセスしてみる

curl -v http://192.168.49.2:31317/indexes/movies/documents
*   Trying 192.168.49.2:31317...
* Connected to 192.168.49.2 (192.168.49.2) port 31317 (#0)
> GET /indexes/movies/documents HTTP/1.1
> Host: 192.168.49.2:31317
> User-Agent: curl/7.73.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 Not Found
< content-length: 168
< content-type: application/json
< date: Tue, 12 Oct 2021 14:11:08 GMT
<
* Connection #0 to host 192.168.49.2 left intact
{"message":"Index movies not found","errorCode":"index_not_found","errorType":"invalid_request_error","errorLink":"https://docs.meilisearch.com/errors#index_not_found"}%                                                                                         

ログを見て0と1にアクセスされていることを確認しながらやってみましたが
ずっと 404 でした

PVCを設定してみる(1台)

まず1台に戻してみます

values.yaml
replicaCount: 1
service:
  type: NodePort
❯ kubectl get po
NAME            READY   STATUS    RESTARTS   AGE
meilisearch-0   1/1     Running   0          9m52s

1台になりました
pvcを設定してみます

values.yaml
replicaCount: 1
service:
  type: NodePort
persistence:
  enabled: true
❯ kubectl get pvc
NAME          STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
meilisearch   Bound    pvc-26f748a1-e02c-49d0-87ef-439be1b73fa7   10Gi       RWO            standard       4s

❯ kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                 STORAGECLASS   REASON   AGE
pvc-26f748a1-e02c-49d0-87ef-439be1b73fa7   10Gi       RWO            Delete           Bound    default/meilisearch   standard                10s

❯ kubectl describe sts meilisearch
Name:               meilisearch
Namespace:          default
CreationTimestamp:  Sat, 09 Oct 2021 01:25:09 +0900
Selector:           app.kubernetes.io/instance=meilisearch,app.kubernetes.io/name=meilisearch
Labels:             app.kubernetes.io/instance=meilisearch
                    app.kubernetes.io/managed-by=Helm
                    app.kubernetes.io/name=meilisearch
                    helm.sh/chart=meilisearch-0.1.18
Annotations:        meta.helm.sh/release-name: meilisearch
                    meta.helm.sh/release-namespace: default
Replicas:           1 desired | 1 total
Update Strategy:    RollingUpdate
  Partition:        0
Pods Status:        1 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:           app.kubernetes.io/instance=meilisearch
                    app.kubernetes.io/name=meilisearch
  Annotations:      kubectl.kubernetes.io/restartedAt: 2021-10-12T23:02:13+09:00
  Service Account:  meilisearch
  Containers:
   meilisearch:
    Image:      getmeili/meilisearch:v0.23.0
    Port:       7700/TCP
    Host Port:  0/TCP
    Liveness:   http-get http://:http/health delay=60s timeout=1s period=60s #success=1 #failure=3
    Readiness:  http-get http://:http/health delay=60s timeout=1s period=60s #success=1 #failure=3
    Environment Variables from:
      meilisearch-environment  ConfigMap  Optional: false
    Environment:               <none>
    Mounts:
      /data.ms from data (rw)
  Volumes:
   data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  meilisearch
    ReadOnly:   false
Volume Claims:  <none>
Events:
  Type    Reason            Age                  From                    Message
  ----    ------            ----                 ----                    -------
  Normal  SuccessfulCreate  15m (x2 over 40m)    statefulset-controller  create Pod meilisearch-1 in StatefulSet meilisearch successful
  Normal  SuccessfulDelete  4m14s (x2 over 15m)  statefulset-controller  delete Pod meilisearch-1 in StatefulSet meilisearch successful
  Normal  SuccessfulDelete  49s (x3 over 49m)    statefulset-controller  delete Pod meilisearch-0 in StatefulSet meilisearch successful
  Normal  SuccessfulCreate  48s (x3 over 49m)    statefulset-controller  create Pod meilisearch-0 in StatefulSet meilisearch successful

できてそうですね

データを入れてみます

curl \
  -H "Content-Type: application/json" \
  -X POST 'http://192.168.49.2:31317/indexes/movies/documents' \
  --data '[{
      "id": 287947,
      "title": "Shazam",
      "poster": "https://image.tmdb.org/t/p/w1280/xnopI5Xtky18MPhK40cZAGAOVeV.jpg",
      "overview": "A boy is given the ability to become an adult superhero in times of need with a single magic word.",
      "release_date": "2019-03-23"
  }]'
{"updateId":0}%                                                                                                   
curl -v http://192.168.49.2:31317/indexes/movies/documents
*   Trying 192.168.49.2:31317...
* Connected to 192.168.49.2 (192.168.49.2) port 31317 (#0)
> GET /indexes/movies/documents HTTP/1.1
> Host: 192.168.49.2:31317
> User-Agent: curl/7.73.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-length: 248
< content-type: application/json
< date: Tue, 12 Oct 2021 14:19:41 GMT
<
* Connection #0 to host 192.168.49.2 left intact
[{"id":287947,"title":"Shazam","poster":"https://image.tmdb.org/t/p/w1280/xnopI5Xtky18MPhK40cZAGAOVeV.jpg","overview":"A boy is given the ability to become an adult superhero in times of need with a single magic word.","release_date":"2019-03-23"}]%                                                                   

取れました。
再起動してみます。

❯ kubectl rollout restart statefulset meilisearch
...
❯ kubectl get po
NAME            READY   STATUS    RESTARTS   AGE
meilisearch-0   1/1     Running   0          111s
curl -v http://192.168.49.2:31317/indexes/movies/documents
*   Trying 192.168.49.2:31317...
* Connected to 192.168.49.2 (192.168.49.2) port 31317 (#0)
> GET /indexes/movies/documents HTTP/1.1
> Host: 192.168.49.2:31317
> User-Agent: curl/7.73.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-length: 248
< content-type: application/json
< date: Tue, 12 Oct 2021 14:21:48 GMT
<
* Connection #0 to host 192.168.49.2 left intact
[{"id":287947,"title":"Shazam","poster":"https://image.tmdb.org/t/p/w1280/xnopI5Xtky18MPhK40cZAGAOVeV.jpg","overview":"A boy is given the ability to become an adult superhero in times of need with a single magic word.","release_date":"2019-03-23"}]%                                                                   

ちゃんとデータが取得できました

冗長化してみる(PVCあり2台)

1台起動してデータを入れた状態で2台目を起動するとどうなるのかやってみます。

values.yaml
replicaCount: 2
service:
  type: NodePort
persistence:
  enabled: true
❯ kubectl logs -f meilisearch-1

888b     d888          d8b 888 d8b  .d8888b.                                    888
8888b   d8888          Y8P 888 Y8P d88P  Y88b                                   888
88888b.d88888              888     Y88b.                                        888
888Y88888P888  .d88b.  888 888 888  "Y888b.    .d88b.   8888b.  888d888 .d8888b 88888b.
888 Y888P 888 d8P  Y8b 888 888 888     "Y88b. d8P  Y8b     "88b 888P"  d88P"    888 "88b
888  Y8P  888 88888888 888 888 888       "888 88888888 .d888888 888    888      888  888
888   "   888 Y8b.     888 888 888 Y88b  d88P Y8b.     888  888 888    Y88b.    888  888
888       888  "Y8888  888 888 888  "Y8888P"   "Y8888  "Y888888 888     "Y8888P 888  888

Database path:          "./data.ms"
Server listening on:    "http://0.0.0.0:7700"
Environment:            "development"
Commit SHA:             "3172c96042747e1ec3d27666b2bbc5ca507c56a7"
Commit date:            "2021-10-11T14:27:46+00:00"
Package version:        "0.23.0"
Anonymous telemetry:    "Disabled"

No master key found; The server will accept unidentified requests. If you need some protection in development mode, please export a key: export MEILI_MASTER_KEY=xxx

Documentation:          https://docs.meilisearch.com
Source code:            https://github.com/meilisearch/meilisearch
Contact:                https://docs.meilisearch.com/resources/contact.html or bonjour@meilisearch.com

[2021-10-12T14:25:00Z INFO  actix_server::builder] Starting 24 workers
[2021-10-12T14:25:00Z INFO  actix_server::builder] Starting "actix-web-service-0.0.0.0:7700" service on 0.0.0.0:7700
[2021-10-12T14:25:00Z ERROR meilisearch_lib::index_controller::updates::store] Fatal error while processing an update that requires the update store to shutdown: Internal error: Resource temporarily unavailable (os error 11)
[2021-10-12T14:25:00Z ERROR meilisearch_lib::index_controller::updates::store] Update store loop exited.

なんかエラーが出てますね。

[2021-10-12T14:26:20Z INFO  actix_web::middleware::logger] 172.17.0.1:58946 "GET /health HTTP/1.1" 200 22 "-" "kube-probe/1.22" 0.000051

しばらくして、health checkが通って起動しました

❯ kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                 STORAGECLASS   REASON   AGE
pvc-26f748a1-e02c-49d0-87ef-439be1b73fa7   10Gi       RWO            Delete           Bound    default/meilisearch   standard                11m


❯ kubectl get pvc
NAME          STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
meilisearch   Bound    pvc-26f748a1-e02c-49d0-87ef-439be1b73fa7   10Gi       RWO            standard       11m

アクセスしてみます

[2021-10-12T14:28:44Z INFO  actix_web::middleware::logger] 172.17.0.1:24365 "GET /indexes/movies/documents HTTP/1.1" 500 187 "-" "curl/7.73.0" 0.000374
[2021-10-12T14:28:47Z INFO  actix_web::middleware::logger] 172.17.0.1:37274 "GET /indexes/movies/documents HTTP/1.1" 500 187 "-" "curl/7.73.0" 0.000341
[2021-10-12T14:28:48Z INFO  actix_web::middleware::logger] 172.17.0.1:45434 "GET /indexes/movies/documents HTTP/1.1" 500 187 "-" "curl/7.73.0" 0.000276
[2021-10-12T14:28:49Z INFO  actix_web::middleware::logger] 172.17.0.1:28171 "GET /indexes/movies/documents HTTP/1.1" 500 187 "-" "curl/7.73.0" 0.000261
[2021-10-12T14:28:49Z INFO  actix_web::middleware::logger] 172.17.0.1:52792 "GET /indexes/movies/documents HTTP/1.1" 500 187 "-" "curl/7.73.0" 0.000224
[2021-10-12T14:28:50Z INFO  actix_web::middleware::logger] 172.17.0.1:51697 "GET /indexes/movies/documents HTTP/1.1" 500 187 "-" "curl/7.73.0" 0.000226
[2021-10-12T14:28:51Z INFO  actix_web::middleware::logger] 172.17.0.1:52843 "GET /indexes/movies/documents HTTP/1.1" 500 187 "-" "curl/7.73.0" 0.000310

meilisearch-1 の方にアクセスされた場合は 500 になってしまいました。

curl -v http://192.168.49.2:31317/indexes/movies/documents
*   Trying 192.168.49.2:31317...
* Connected to 192.168.49.2 (192.168.49.2) port 31317 (#0)
> GET /indexes/movies/documents HTTP/1.1
> Host: 192.168.49.2:31317
> User-Agent: curl/7.73.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 500 Internal Server Error
< content-length: 187
< content-type: application/json
< date: Tue, 12 Oct 2021 14:35:09 GMT
<
* Connection #0 to host 192.168.49.2 left intact
{"message":"Internal Error: Resource temporarily unavailable (os error 11)","errorCode":"internal","errorType":"internal_error","errorLink":"https://docs.meilisearch.com/errors#internal"}%              

やっぱり同じVolumeを参照しているのがまずいような? 😣
ただIssueも無いし、使い方がおかしいのか? ※後で調べる

MEILI_MASTER_KEY

meilisearchの MEILI_ENVproduction に設定する場合 MEILI_MASTER_KEY が必要になります。

https://github.com/meilisearch/meilisearch-kubernetes/blob/main/charts/meilisearch/README.md

どうやらSecretが自動的に作成されるようです。
やってみます。

values.yaml
replicaCount: 2
service:
  type: NodePort
persistence:
  enabled: true
environment:
  MEILI_ENV: production
❯ helm template meilisearch/meilisearch -f values.yaml
...
# Source: meilisearch/templates/master-key-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: RELEASE-NAME-meilisearch-master-key
  labels:
    app.kubernetes.io/name: meilisearch
    helm.sh/chart: meilisearch-0.1.18
    app.kubernetes.io/instance: RELEASE-NAME
    app.kubernetes.io/managed-by: Helm
data:
  MEILI_MASTER_KEY: aTNrV002UzBseVRPSzREQldHN0s=
...

たしかに追加されました。

https://github.com/meilisearch/meilisearch-kubernetes/blob/b5faa95143df897bd786105b31c1e20dcce83022/charts/meilisearch/templates/master-key-secret.yaml#L1-L22
  • meilisearch-master-key という名前のSecretを探しにいく
  • あれば、その中の MEILI_MASTER_KEY を使ってSecretを生成する
  • なければ、 {{ randAlphaNum 20 | b64enc }} でランダムな値を生成する

ということをやってそう。

ではSecretを事前に作っておけば、値を設定できる?
やってみます。

secret-master-key.yaml
apiVersion: v1
kind: Secret
metadata:
  name: meilisearch-master-key
type: Opaque
stringData:
  MEILI_MASTER_KEY: apikey
❯ kubectl apply -f secret-master-key.yaml
secret/meilisearch-master-key created

❯ kubectl describe secret meilisearch-master-key
Name:         meilisearch-master-key
Namespace:    default
Labels:       <none>
Annotations:  <none>

Type:  Opaque

Data
====
MEILI_MASTER_KEY:  6 bytes
values.yaml
replicaCount: 1
service:
  type: NodePort
persistence:
  enabled: true
environment:
  MEILI_ENV: production
> ./upgrade.sh
...
Error: UPGRADE FAILED: rendered manifests contain a resource that already exists. Unable to continue with update: Secret "meilisearch-master-key" in namespace "default" exists and cannot be imported into the current release: invalid ownership metadata; label validation error: missing key "app.kubernetes.io/managed-by": must be set to "Helm"; annotation validation error: missing key "meta.helm.sh/release-name": must be set to "meilisearch"; annotation validation error: missing key "meta.helm.sh/release-namespace": must be set to "default"

エラーになりました。
なるほど、annotationとlabelsを設定しておかないとHelmから上書きできないんですね。
直してみます。

secret-master-key.yaml
apiVersion: v1
kind: Secret
metadata:
  name: meilisearch-master-key
  annotations:
    meta.helm.sh/release-name: meilisearch
    meta.helm.sh/release-namespace: default
  labels:
    app.kubernetes.io/managed-by: Helm
type: Opaque
stringData:
  MEILI_MASTER_KEY: apikey
❯ kubectl apply -f secret-master-key.yaml
...

❯ ./upgrade.sh
"meilisearch" already exists with the same configuration, skipping
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "meilisearch" chart repository
...Successfully got an update from the "bitnami" chart repository
Update Complete. ⎈Happy Helming!⎈
Release "meilisearch" has been upgraded. Happy Helming!
NAME: meilisearch
LAST DEPLOYED: Wed Oct 13 00:08:20 2021
NAMESPACE: default
STATUS: deployed
REVISION: 9
NOTES:
1. Get the application URL by running these commands:
  export NODE_PORT=$(kubectl get --namespace default -o jsonpath="{.spec.ports[0].nodePort}" services meilisearch)
  export NODE_IP=$(kubectl get nodes --namespace default -o jsonpath="{.items[0].status.addresses[0].address}")
  echo http://$NODE_IP:$NODE_PORT

デプロイできました。

❯ kubectl describe secret meilisearch-master-key
Name:         meilisearch-master-key
Namespace:    default
Labels:       app.kubernetes.io/instance=meilisearch
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=meilisearch
              helm.sh/chart=meilisearch-0.1.18
Annotations:  meta.helm.sh/release-name: meilisearch
              meta.helm.sh/release-namespace: default

Type:  Opaque

Data
====
MEILI_MASTER_KEY:  6 bytes

labelsが増えているのでhelmによって上書きされたっぽいですね。
※ただ、これやるとExternalSecretみたいなのと併用した時に相性悪いかもしれないので要調査

curl -v -H "X-Meili-API-Key: apikey" http://192.168.49.2:31317/indexes/movies/documents
*   Trying 192.168.49.2:31317...
* Connected to 192.168.49.2 (192.168.49.2) port 31317 (#0)
> GET /indexes/movies/documents HTTP/1.1
> Host: 192.168.49.2:31317
> User-Agent: curl/7.73.0
> Accept: */*
> X-Meili-API-Key: apikey
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-length: 248
< content-type: application/json
< date: Tue, 12 Oct 2021 15:14:25 GMT
<
* Connection #0 to host 192.168.49.2 left intact
[{"id":287947,"title":"Shazam","poster":"https://image.tmdb.org/t/p/w1280/xnopI5Xtky18MPhK40cZAGAOVeV.jpg","overview":"A boy is given the ability to become an adult superhero in times of need with a single magic word.","release_date":"2019-03-23"}]%                                                                   

X-Meili-API-Key ヘッダにkeyを渡してアクセスできました。

まとめ

長くなってきたので、今回はこの辺で...

  • 冗長化&PVCでデータを永続化したい場合がまだ上手く行ってない
    • 次回以降で調査するかも
  • master-keyの管理方法はSecretManagerで管理してExternalSecretで取得したい
    • helmによってSecretが生成されてしまうので、その辺どうするかは要検討
    • そもそも諦めてenvFromのsecretRefで他のpodから meilisearch-master-key を参照するのがいいのか?

Discussion

ログインするとコメントできます