🔎
MeiliSearch使ってみる kubernetes編
使い方はいくらかわかってきたので、運用面を考えてみます
前回までの話
運用
- k8s内で使いたい
- helm chartがあるのでhelmでデプロイ＆バージョン管理する
- 冗長化したい
- 冗長化した場合永続化されたデータはどうやってどこに持つのだろうか？
- そもそもできるのか
環境
- OS: Ubuntu 20.04
- MeiliSearch: 0.22.0
- meilisearch/meilisearch-kubernetes: 0.1.18
- minikube: v1.23.2
- kubectl: 1.22.2
準備
minikube, kubectlのインストール
参考にさせていただきました 🙏
minikubeの起動
過去の設定が悪さしているっぽくエラーで起動できなかったので一回削除しました
❯ minikube start --kubernetes-version=v1.22.2
😄 Ubuntu 20.04 上の minikube v1.23.2
✨ dockerドライバーが自動的に選択されました。他の選択肢: virtualbox, none, ssh
👍 コントロールプレーンのノード minikube を minikube 上で起動しています
🚜 イメージを Pull しています...
🔥 docker container (CPUs=2, Memory=16000MB) を作成しています...
🐳 Docker 20.10.8 で Kubernetes v1.22.2 を準備しています...
▪ 証明書と鍵を作成しています...
▪ Control Plane を起動しています...
▪ RBAC のルールを設定中です.../ E1009 01:11:25.835511 66512 start.go:142] Unable to scale down deployment "coredns" in namespace "kube-system" to 1 replica: timed out waiting for the condition
🔎 Kubernetes コンポーネントを検証しています...
▪ イメージ gcr.io/k8s-minikube/storage-provisioner:v5 を使用しています
❗ 'default-storageclass' を有効にする際にエラーが発生しました。running callbacks: [Error making standard the default storage class: Error listing StorageClasses: Unauthorized]
🌟 有効なアドオン: storage-provisioner
❌ Exiting due to K8S_UNHEALTHY_CONTROL_PLANE: wait 6m0s for node: wait for healthy API server: controlPlane never updated to v1.22.2
💡 提案: Control Plane could not update, try minikube delete --all --purge
🍿 Related issue: https://github.com/kubernetes/minikube/issues/11417
╭───────────────────────────────────────────────────────────────────────────────────────────╮
│ │
│ 😿 If the above advice does not help, please let us know: │
│ 👉 https://github.com/kubernetes/minikube/issues/new/choose │
│ │
│ Please run `minikube logs --file=logs.txt` and attach logs.txt to the GitHub issue. │
│ │
╰───────────────────────────────────────────────────────────────────────────────────────────╯
❯ minikube delete --all --purge
🔥 docker の「minikube」を削除しています...
🔥 /home/xxxx/.minikube/machines/minikube を削除しています...
💀 クラスタ "minikube" の全てのトレースを削除しました。
🔥 全てのプロファイルの削除に成功しました
💀 Successfully purged minikube directory located at - [/home/xxxx/.minikube]
❯ minikube start --kubernetes-version=v1.22.2
😄 Ubuntu 20.04 上の minikube v1.23.2
✨ dockerドライバーが自動的に選択されました。他の選択肢: virtualbox, none, ssh
👍 コントロールプレーンのノード minikube を minikube 上で起動しています
🚜 イメージを Pull しています...
💾 Kubernetes v1.22.2 のダウンロードの準備をしています
> preloaded-images-k8s-v13-v1...: 511.84 MiB / 511.84 MiB 100.00% 76.70 Mi
🔥 docker container (CPUs=2, Memory=16000MB) を作成しています...
🐳 Docker 20.10.8 で Kubernetes v1.22.2 を準備しています...
▪ 証明書と鍵を作成しています...
▪ Control Plane を起動しています...
▪ RBAC のルールを設定中です...
🔎 Kubernetes コンポーネントを検証しています...
▪ イメージ gcr.io/k8s-minikube/storage-provisioner:v5 を使用しています
🌟 有効なアドオン: default-storageclass, storage-provisioner
🏄 完了しました！ kubectl が「"minikube"」クラスタと「"default"」ネームスペースを使用するよう構成されました
❯ minikube status
minikube
type: Control Plane
host: Running
kubelet: Running
apiserver: Running
kubeconfig: Configured
❯ kubectl get po
No resources found in default namespace.
大丈夫そうです。
Helmのインストール
MeiliSearchのインストール
READMEを見て進めます。
Helm repo
❯ helm repo add meilisearch https://meilisearch.github.io/meilisearch-kubernetes
"meilisearch" has been added to your repositories
install
❯ helm upgrade -i meilisearch meilisearch/meilisearch
Release "meilisearch" does not exist. Installing it now.
NAME: meilisearch
LAST DEPLOYED: Sat Oct 9 01:25:08 2021
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTES:
1. Get the application URL by running these commands:
kubectl port-forward --namespace default svc/meilisearch 7700:7700 &
echo "Visit http://127.0.0.1:7700 to use your application"
❯ helm list
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
meilisearch default 1 2021-10-09 01:25:08.954581332 +0900 JST deployed meilisearch-0.1.17 v0.22.0
❯ kubectl get po
NAME READY STATUS RESTARTS AGE
meilisearch-0 0/1 Running 0 53s
起動したっぽいですね
アクセスしてみる
kubectl port-forward でアクセスしてみます
❯ kubectl port-forward svc/meilisearch 7700:7700
Forwarding from 127.0.0.1:7700 -> 7700
Forwarding from [::1]:7700 -> 7700
Handling connection for 7700
❯ curl -v http://localhost:7700
* Trying 127.0.0.1:7700...
* Connected to localhost (127.0.0.1) port 7700 (#0)
> GET / HTTP/1.1
> Host: localhost:7700
> User-Agent: curl/7.73.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-length: 2191
< content-type: text/html
< date: Fri, 08 Oct 2021 16:31:39 GMT
<
<!doctype html><html lang="en"><head><meta charset="utf-8"/><link rel="icon" href="/favicon.ico"/><meta name="viewport" content="width=device-width,initial-scale=1"/><meta name="theme-color" content="#000000"/><meta name="description" content="Dashboard to test MeiliSearch's search engine"/><link rel="apple-touch-icon" href="/logo.png"/><link rel="manifest" href="/manifest.json"/><title>Mini-dashboard | MeiliSearch</title></head><body><noscript>You need to enable JavaScript to run this app.</noscript><div id="root"></div><script>!function(e){function r(r){for(var n,i,a=r[0],l=r[1],f=r[2],c=0,s=[];c<a.length;c++)i=a[c],Object.prototype.hasOwnProperty.call(o,i)&&o[i]&&s.push(o[i][0]),o[i]=0;for(n in l)Object.prototype.hasOwnProperty.call(l,n)&&(e[n]=l[n]);for(p&&p(r);s.length;)s.shift()();return u.push.apply(u,f||[]),t()}function t(){for(var e,r=0;r<u.length;r++){for(var t=u[r],n=!0,a=1;a<t.length;a++){var l=t[a];0!==o[l]&&(n=!1)}n&&(u.splice(r--,1),e=i(i.s=t[0]))}return e}var n={},o={1:0},u=[];function i(r){if(n[r])return n[r].exports;var t=n[r]={i:r,l:!1,exports:{}};return e[r].call(t.exports,t,t.exports,i),t.l=!0,t.exports}i.m=e,i.c=n,i.d=function(e,r,t){i.o(e,r)||Object.defineProperty(e,r,{enumerable:!0,get:t})},i.r=function(e){"undefined"!=typeof Symbol&&Symbol.toStringTag&&Object.defineProperty(e,Symbol.toStringTag,{value:"Module"}),Object.defineProperty(e,"__esModule",{value:!0})},i.t=function(e,r){if(1&r&&(e=i(e)),8&r)return e;if(4&r&&"object"==typeof e&&e&&e.__esModule)return e;var t=Object.create(null);if(i.r(t),Object.defineProperty(t,"default",{enumerable:!0,value:e}),2&r&&"string"!=typeof e)for(var n in e)i.d(t,n,function(r){return e[r]}.bind(null,n));return t},i.n=function(e){var r=e&&e.__esModule?function(){return e.default}:function(){return e};return i.d(r,"a",r),r},i.o=function(e,r){return Object.prototype.hasOwnProperty.call(e,r)},i.p="/";var a=this["webpackJsonpmini-dashboard"]=this["webpackJsonpmini-dashboard"]||[],l=a.push.bind(a);a.push=r,a=a.slice();for(var f=0;f<a.length;f++)r(a[f]);var p=l* Connection #0 to host localhost left intact
;t()}([])</script><script src="/static/js/2.b22bb78b.chunk.js"></script><script src="/static/js/main.81816167.chunk.js"></script></body></html>%
❯ curl -v http://localhost:7700/indexes
* Trying 127.0.0.1:7700...
* Connected to localhost (127.0.0.1) port 7700 (#0)
> GET /indexes HTTP/1.1
> Host: localhost:7700
> User-Agent: curl/7.73.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-length: 2
< content-type: application/json
< date: Fri, 08 Oct 2021 16:34:10 GMT
<
* Connection #0 to host localhost left intact
[]%
アクセスできました
冗長化したい
helm chartのテンプレートを確認する
まずは、helm chartの中身を確認してどういう構成になっているのか見てみます
❯ helm template meilisearch/meilisearch
---
# Source: meilisearch/templates/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: RELEASE-NAME-meilisearch
labels:
app.kubernetes.io/name: meilisearch
helm.sh/chart: meilisearch-0.1.17
app.kubernetes.io/instance: RELEASE-NAME
app.kubernetes.io/managed-by: Helm
---
# Source: meilisearch/templates/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: RELEASE-NAME-meilisearch-environment
labels:
app.kubernetes.io/name: meilisearch
helm.sh/chart: meilisearch-0.1.17
app.kubernetes.io/instance: RELEASE-NAME
app.kubernetes.io/managed-by: Helm
data:
MEILI_ENV: "development"
MEILI_NO_ANALYTICS: "true"
---
# Source: meilisearch/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
name: RELEASE-NAME-meilisearch
labels:
app.kubernetes.io/name: meilisearch
helm.sh/chart: meilisearch-0.1.17
app.kubernetes.io/instance: RELEASE-NAME
app.kubernetes.io/managed-by: Helm
spec:
type: ClusterIP
ports:
- port: 7700
targetPort: http
protocol: TCP
name: http
selector:
app.kubernetes.io/name: meilisearch
app.kubernetes.io/instance: RELEASE-NAME
---
# Source: meilisearch/templates/statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: RELEASE-NAME-meilisearch
labels:
app.kubernetes.io/name: meilisearch
helm.sh/chart: meilisearch-0.1.17
app.kubernetes.io/instance: RELEASE-NAME
app.kubernetes.io/managed-by: Helm
spec:
replicas: 1
serviceName: RELEASE-NAME-meilisearch
selector:
matchLabels:
app.kubernetes.io/name: meilisearch
app.kubernetes.io/instance: RELEASE-NAME
template:
metadata:
labels:
app.kubernetes.io/name: meilisearch
app.kubernetes.io/instance: RELEASE-NAME
spec:
serviceAccountName: RELEASE-NAME-meilisearch
containers:
- name: meilisearch
image: "getmeili/meilisearch:v0.22.0"
imagePullPolicy: IfNotPresent
envFrom:
- configMapRef:
name: RELEASE-NAME-meilisearch-environment
ports:
- name: http
containerPort: 7700
protocol: TCP
livenessProbe:
httpGet:
path: /health
port: http
periodSeconds: 60
initialDelaySeconds: 60
readinessProbe:
httpGet:
path: /health
port: http
periodSeconds: 60
initialDelaySeconds: 60
resources:
{}
---
# Source: meilisearch/templates/tests/test-connection.yaml
apiVersion: v1
kind: Pod
metadata:
name: release-name-meilisearch-test-connection
labels:
app.kubernetes.io/name: meilisearch
helm.sh/chart: meilisearch-0.1.17
app.kubernetes.io/instance: RELEASE-NAME
app.kubernetes.io/managed-by: Helm
annotations:
"helm.sh/hook": test-success
spec:
containers:
- name: wget
image: busybox
command: ['wget']
args: ['RELEASE-NAME-meilisearch:7700']
restartPolicy: Never
なるほど StatefulSet で起動されているんですね。
この辺を見ると、persistence.enabledをtrueにすればpvcを作ってデータを保存できそうな感じですかね
オートスケールはできなそうですが、↑ここでreplicasを指定できるので常時２台以上起動するようなことはできそうですかね。
自前でHPAを書けばオートスケールできるだろうか？ StatefulSetにHPAを設定したことがないので後で確認する
kubectlからhelmでデプロイされたリソースを確認してみます。
❯ kubectl get po
NAME READY STATUS RESTARTS AGE
meilisearch-0 1/1 Running 1 (3d ago) 3d20h
❯ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 3d20h
meilisearch ClusterIP 10.107.240.7 <none> 7700/TCP 3d20h
❯ kubectl get statefulset
NAME READY AGE
meilisearch 1/1 3d20h
❯ kubectl get hpa
No resources found in default namespace.
❯ kubectl get cm
NAME DATA AGE
kube-root-ca.crt 1 3d20h
meilisearch-environment 2 3d20h
❯ kubectl get sa
NAME SECRETS AGE
default 1 3d20h
meilisearch 1 3d20h
helm chartのvaluesを確認する
なにを拡張できるのか、helmのvaluesを見てみます。
❯ helm show values meilisearch/meilisearch
# Default values for <CHARTNAME>.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.
replicaCount: 1
readinessProbe:
periodSeconds: 60
InitialDelaySeconds: 60
livenessProbe:
periodSeconds: 60
InitialDelaySeconds: 60
image:
repository: getmeili/meilisearch
tag: v0.22.0
pullPolicy: IfNotPresent
nameOverride: ""
fullnameOverride: ""
# Environment loaded into the configMap
environment:
MEILI_NO_ANALYTICS: true
MEILI_ENV: development
# For production deployment, the environment MEILI_MASTER_KEY is required.
# If MEILI_ENV is set to "production" without setting MEILI_MASTER_KEY, this
# chart will automatically create a secure MEILI_MASTER_KEY and push it as a
# secret. Otherwise the below value of MEILI_MASTER_KEY will be used instead.
# MEILI_MASTER_KEY:
podAnnotations: {}
customLabels: {}
service:
type: ClusterIP
port: 7700
container:
containerPort: 7700
ingress:
enabled: false
annotations: {}
# kubernetes.io/ingress.class: nginx
path: /
hosts:
- meilisearch-example.local
tls: []
# - secretName: chart-example-tls
# hosts:
# - chart-example.local
persistence:
enabled: false
accessMode: ReadWriteOnce
## Persistent Volume Storage Class
## If defined, storageClassName: <storageClass>
## If set to "-", storageClassName: "", which disables dynamic provisioning
## If undefined (the default) or set to null, no storageClassName spec is
## set, choosing the default provisioner. (gp2 on AWS, standard on
## GKE, AWS & OpenStack)
##
# storageClass: "-"
size: 10Gi
volume:
name: data
mountPath: /data.ms
resources: {}
# We usually recommend not to specify default resources and to leave this as a conscious
# choice for the user. This also increases chances charts run on environments with little
# resources, such as Minikube. If you do want to specify resources, uncomment the following
# lines, adjust them as necessary, and remove the curly braces after 'resources:'.
# limits:
# cpu: 100m
# memory: 128Mi
# requests:
# cpu: 100m
# memory: 128Mi
nodeSelector: {}
tolerations: []
affinity: {}
-
persistence- PVCを作る設定
-
nameOverride- https://github.com/meilisearch/meilisearch-kubernetes/blob/b5faa95143df897bd786105b31c1e20dcce83022/charts/meilisearch/templates/_helpers.tpl#L5-L7
- labelの
app.kubernetes.io/nameなどで使われている
-
fullnameOverride
valuesを独自に設定してみる(PVCなし)
まず事前に多少データを登録しておきます。
❯ curl \
-H "Content-Type: application/json" \
-X POST 'http://localhost:7700/indexes/movies/documents' \
--data '[{
"id": 287947,
"title": "Shazam",
"poster": "https://image.tmdb.org/t/p/w1280/xnopI5Xtky18MPhK40cZAGAOVeV.jpg",
"overview": "A boy is given the ability to become an adult superhero in times of need with a single magic word.",
"release_date": "2019-03-23"
}]'
{"updateId":0}% ```
```shell
❯ curl -v http://localhost:7700/indexes/movies/documents
* Trying 127.0.0.1:7700...
* Connected to localhost (127.0.0.1) port 7700 (#0)
> GET /indexes/movies/documents HTTP/1.1
> Host: localhost:7700
> User-Agent: curl/7.73.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-length: 248
< content-type: application/json
< date: Tue, 12 Oct 2021 13:34:36 GMT
<
* Connection #0 to host localhost left intact
[{"id":287947,"title":"Shazam","poster":"https://image.tmdb.org/t/p/w1280/xnopI5Xtky18MPhK40cZAGAOVeV.jpg","overview":"A boy is given the ability to become an adult superhero in times of need with a single magic word.","release_date":"2019-03-23"}]%
とれました。
ではvaluesを設定して、2台起動してみます。
values.yaml
replicaCount: 2
upgrade.sh
#!/bin/bash -e
helm repo add meilisearch https://meilisearch.github.io/meilisearch-kubernetes
helm repo update
helm upgrade -i meilisearch meilisearch/meilisearch --version 0.1.18 -f values.yaml
❯ kubectl get po
NAME READY STATUS RESTARTS AGE
meilisearch-0 1/1 Running 0 10m
meilisearch-1 1/1 Running 0 99s
２台起動しました。
アクセスしてみます。
❯ kubectl port-forward svc/meilisearch 7700:7700
❯ watch -n 0.5 "curl -v http://localhost:7700/indexes/movies/documents"
...
meilisearch-0 のログ
❯ kubectl logs -f meilisearch-0
888b d888 d8b 888 d8b .d8888b. 888
8888b d8888 Y8P 888 Y8P d88P Y88b 888
88888b.d88888 888 Y88b. 888
888Y88888P888 .d88b. 888 888 888 "Y888b. .d88b. 8888b. 888d888 .d8888b 88888b.
888 Y888P 888 d8P Y8b 888 888 888 "Y88b. d8P Y8b "88b 888P" d88P" 888 "88b
888 Y8P 888 88888888 888 888 888 "888 88888888 .d888888 888 888 888 888
888 " 888 Y8b. 888 888 888 Y88b d88P Y8b. 888 888 888 Y88b. 888 888
888 888 "Y8888 888 888 888 "Y8888P" "Y8888 "Y888888 888 "Y8888P 888 888
Database path: "./data.ms"
Server listening on: "http://0.0.0.0:7700"
Environment: "development"
Commit SHA: "3172c96042747e1ec3d27666b2bbc5ca507c56a7"
Commit date: "2021-10-11T14:27:46+00:00"
Package version: "0.23.0"
Anonymous telemetry: "Disabled"
No master key found; The server will accept unidentified requests. If you need some protection in development mode, please export a key: export MEILI_MASTER_KEY=xxx
Documentation: https://docs.meilisearch.com
Source code: https://github.com/meilisearch/meilisearch
Contact: https://docs.meilisearch.com/resources/contact.html or bonjour@meilisearch.com
[2021-10-12T13:28:22Z INFO actix_server::builder] Starting 24 workers
[2021-10-12T13:28:22Z INFO actix_server::builder] Starting "actix-web-service-0.0.0.0:7700" service on 0.0.0.0:7700
[2021-10-12T13:29:28Z INFO actix_web::middleware::logger] 127.0.0.1:33672 "GET / HTTP/1.1" 200 2191 "-" "curl/7.73.0" 0.000060
[2021-10-12T13:29:32Z INFO actix_web::middleware::logger] 172.17.0.1:45660 "GET /health HTTP/1.1" 200 22 "-" "kube-probe/1.22" 0.000052
...
[2021-10-12T13:39:20Z INFO actix_web::middleware::logger] 127.0.0.1:37512 "GET /indexes/movies/documents HTTP/1.1" 200 248 "-" "curl/7.73.0" 0.000330
[2021-10-12T13:39:20Z INFO actix_web::middleware::logger] 127.0.0.1:37516 "GET /indexes/movies/documents HTTP/1.1" 200 248 "-" "curl/7.73.0" 0.000311
[2021-10-12T13:39:21Z INFO actix_web::middleware::logger] 127.0.0.1:37524 "GET /indexes/movies/documents HTTP/1.1" 200 248 "-" "curl/7.73.0" 0.000412
[2021-10-12T13:39:21Z INFO actix_web::middleware::logger] 127.0.0.1:37528 "GET /indexes/movies/documents HTTP/1.1" 200 248 "-" "curl/7.73.0" 0.000318
[2021-10-12T13:39:22Z INFO actix_web::middleware::logger] 127.0.0.1:37536 "GET /indexes/movies/documents HTTP/1.1" 200 248 "-" "curl/7.73.0" 0.000357
meilisearch-1 のログ
❯ kubectl logs -f meilisearch-1
888b d888 d8b 888 d8b .d8888b. 888
8888b d8888 Y8P 888 Y8P d88P Y88b 888
88888b.d88888 888 Y88b. 888
888Y88888P888 .d88b. 888 888 888 "Y888b. .d88b. 8888b. 888d888 .d8888b 88888b.
888 Y888P 888 d8P Y8b 888 888 888 "Y88b. d8P Y8b "88b 888P" d88P" 888 "88b
888 Y8P 888 88888888 888 888 888 "888 88888888 .d888888 888 888 888 888
888 " 888 Y8b. 888 888 888 Y88b d88P Y8b. 888 888 888 Y88b. 888 888
888 888 "Y8888 888 888 888 "Y8888P" "Y8888 "Y888888 888 "Y8888P 888 888
Database path: "./data.ms"
Server listening on: "http://0.0.0.0:7700"
Environment: "development"
Commit SHA: "3172c96042747e1ec3d27666b2bbc5ca507c56a7"
Commit date: "2021-10-11T14:27:46+00:00"
Package version: "0.23.0"
Anonymous telemetry: "Disabled"
No master key found; The server will accept unidentified requests. If you need some protection in development mode, please export a key: export MEILI_MASTER_KEY=xxx
Documentation: https://docs.meilisearch.com
Source code: https://github.com/meilisearch/meilisearch
Contact: https://docs.meilisearch.com/resources/contact.html or bonjour@meilisearch.com
[2021-10-12T13:37:22Z INFO actix_server::builder] Starting 24 workers
[2021-10-12T13:37:23Z INFO actix_server::builder] Starting "actix-web-service-0.0.0.0:7700" service on 0.0.0.0:7700
[2021-10-12T13:38:22Z INFO actix_web::middleware::logger] 172.17.0.1:38196 "GET /health HTTP/1.1" 200 22 "-" "kube-probe/1.22" 0.000072
[2021-10-12T13:38:22Z INFO actix_web::middleware::logger] 172.17.0.1:38198 "GET /health HTTP/1.1" 200 22 "-" "kube-probe/1.22" 0.000075
[2021-10-12T13:39:22Z INFO actix_web::middleware::logger] 172.17.0.1:38908 "GET /health HTTP/1.1" 200 22 "-" "kube-probe/1.22" 0.000054
[2021-10-12T13:39:22Z INFO actix_web::middleware::logger] 172.17.0.1:38906 "GET /health HTTP/1.1" 200 22 "-" "kube-probe/1.22" 0.000051
[2021-10-12T13:40:22Z INFO actix_web::middleware::logger] 172.17.0.1:39666 "GET /health HTTP/1.1" 200 22 "-" "kube-probe/1.22" 0.000054
[2021-10-12T13:40:22Z INFO actix_web::middleware::logger] 172.17.0.1:39668 "GET /health HTTP/1.1" 200 22 "-" "kube-probe/1.22" 0.000044
...
meilisearch-0 にしかアクセスが無いですね..
サービスに対してフォワードしているわけではないので、フォワードしたポートに何度リクエストを送ってもロードバランスされない。
なるほど、そうなんですね。。
あまりローカルでk8s使わないので、これは知りませんでした。
脱線: minikubeで起動したサービスにアクセスしたい
ノードポート経由で公開されているサービスにアクセスするには、Minikubeを起動してアドレスを取得した後、シェルでこのコマンドを実行してください:
minikube service [-n NAMESPACE] [--url] NAME
なるほど？
❯ minikube service meilisearch
|-----------|-------------|-------------|--------------|
| NAMESPACE | NAME | TARGET PORT | URL |
|-----------|-------------|-------------|--------------|
| default | meilisearch | | No node port |
|-----------|-------------|-------------|--------------|
😿 サービス default/meilisearch は NodePort を持っていません
なるほど？
❯ kubectl describe svc/meilisearch
Name: meilisearch
Namespace: default
Labels: app.kubernetes.io/instance=meilisearch
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=meilisearch
helm.sh/chart=meilisearch-0.1.18
Annotations: meta.helm.sh/release-name: meilisearch
meta.helm.sh/release-namespace: default
Selector: app.kubernetes.io/instance=meilisearch,app.kubernetes.io/name=meilisearch
Type: ClusterIP
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.107.240.7
IPs: 10.107.240.7
Port: http 7700/TCP
TargetPort: http/TCP
Endpoints: 172.17.0.3:7700,172.17.0.4:7700
Session Affinity: None
Events: <none>
たしかに、ClusterIPになってますね。
service.type で変更できそうです。
values.yaml
replicaCount: 2
service:
type: NodePort
❯ minikube service meilisearch
|-----------|-------------|-------------|---------------------------|
| NAMESPACE | NAME | TARGET PORT | URL |
|-----------|-------------|-------------|---------------------------|
| default | meilisearch | http/7700 | http://192.168.49.2:31317 |
|-----------|-------------|-------------|---------------------------|
🎉 Opening service default/meilisearch in default browser...
きました
❯ curl -v http://192.168.49.2:31317/indexes/movies/documents
* Trying 192.168.49.2:31317...
* Connected to 192.168.49.2 (192.168.49.2) port 31317 (#0)
> GET /indexes/movies/documents HTTP/1.1
> Host: 192.168.49.2:31317
> User-Agent: curl/7.73.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 Not Found
< content-length: 168
< content-type: application/json
< date: Tue, 12 Oct 2021 13:59:51 GMT
<
* Connection #0 to host 192.168.49.2 left intact
{"message":"Index movies not found","errorCode":"index_not_found","errorType":"invalid_request_error","errorLink":"https://docs.meilisearch.com/errors#index_not_found"}%
❯ curl -v http://192.168.49.2:31317/indexes/movies/documents
* Trying 192.168.49.2:31317...
* Connected to 192.168.49.2 (192.168.49.2) port 31317 (#0)
> GET /indexes/movies/documents HTTP/1.1
> Host: 192.168.49.2:31317
> User-Agent: curl/7.73.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-length: 248
< content-type: application/json
< date: Tue, 12 Oct 2021 13:59:53 GMT
<
* Connection #0 to host 192.168.49.2 left intact
[{"id":287947,"title":"Shazam","poster":"https://image.tmdb.org/t/p/w1280/xnopI5Xtky18MPhK40cZAGAOVeV.jpg","overview":"A boy is given the ability to become an adult superhero in times of need with a single magic word.","release_date":"2019-03-23"}]%
うん
-
meilisearch-0に行った時はデータが取れて200 -
meilisearch-1に行った時はindexがないので404
になってますね。
OK
再起動してみる
再起動するとデータは消えるのか？
PVCが無いので消えそうな気がする
❯ kubectl rollout restart statefulset meilisearch
❯ kubectl get po
NAME READY STATUS RESTARTS AGE
meilisearch-0 1/1 Running 0 6m11s
meilisearch-1 1/1 Running 0 7m35s
再起動したのでアクセスしてみる
❯ curl -v http://192.168.49.2:31317/indexes/movies/documents
* Trying 192.168.49.2:31317...
* Connected to 192.168.49.2 (192.168.49.2) port 31317 (#0)
> GET /indexes/movies/documents HTTP/1.1
> Host: 192.168.49.2:31317
> User-Agent: curl/7.73.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 Not Found
< content-length: 168
< content-type: application/json
< date: Tue, 12 Oct 2021 14:11:08 GMT
<
* Connection #0 to host 192.168.49.2 left intact
{"message":"Index movies not found","errorCode":"index_not_found","errorType":"invalid_request_error","errorLink":"https://docs.meilisearch.com/errors#index_not_found"}%
ログを見て0と1にアクセスされていることを確認しながらやってみましたが
ずっと 404 でした
PVCを設定してみる（1台)
まず１台に戻してみます
values.yaml
replicaCount: 1
service:
type: NodePort
❯ kubectl get po
NAME READY STATUS RESTARTS AGE
meilisearch-0 1/1 Running 0 9m52s
１台になりました
pvcを設定してみます
values.yaml
replicaCount: 1
service:
type: NodePort
persistence:
enabled: true
❯ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
meilisearch Bound pvc-26f748a1-e02c-49d0-87ef-439be1b73fa7 10Gi RWO standard 4s
❯ kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-26f748a1-e02c-49d0-87ef-439be1b73fa7 10Gi RWO Delete Bound default/meilisearch standard 10s
❯ kubectl describe sts meilisearch
Name: meilisearch
Namespace: default
CreationTimestamp: Sat, 09 Oct 2021 01:25:09 +0900
Selector: app.kubernetes.io/instance=meilisearch,app.kubernetes.io/name=meilisearch
Labels: app.kubernetes.io/instance=meilisearch
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=meilisearch
helm.sh/chart=meilisearch-0.1.18
Annotations: meta.helm.sh/release-name: meilisearch
meta.helm.sh/release-namespace: default
Replicas: 1 desired | 1 total
Update Strategy: RollingUpdate
Partition: 0
Pods Status: 1 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
Labels: app.kubernetes.io/instance=meilisearch
app.kubernetes.io/name=meilisearch
Annotations: kubectl.kubernetes.io/restartedAt: 2021-10-12T23:02:13+09:00
Service Account: meilisearch
Containers:
meilisearch:
Image: getmeili/meilisearch:v0.23.0
Port: 7700/TCP
Host Port: 0/TCP
Liveness: http-get http://:http/health delay=60s timeout=1s period=60s #success=1 #failure=3
Readiness: http-get http://:http/health delay=60s timeout=1s period=60s #success=1 #failure=3
Environment Variables from:
meilisearch-environment ConfigMap Optional: false
Environment: <none>
Mounts:
/data.ms from data (rw)
Volumes:
data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: meilisearch
ReadOnly: false
Volume Claims: <none>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulCreate 15m (x2 over 40m) statefulset-controller create Pod meilisearch-1 in StatefulSet meilisearch successful
Normal SuccessfulDelete 4m14s (x2 over 15m) statefulset-controller delete Pod meilisearch-1 in StatefulSet meilisearch successful
Normal SuccessfulDelete 49s (x3 over 49m) statefulset-controller delete Pod meilisearch-0 in StatefulSet meilisearch successful
Normal SuccessfulCreate 48s (x3 over 49m) statefulset-controller create Pod meilisearch-0 in StatefulSet meilisearch successful
できてそうですね
データを入れてみます
❯ curl \
-H "Content-Type: application/json" \
-X POST 'http://192.168.49.2:31317/indexes/movies/documents' \
--data '[{
"id": 287947,
"title": "Shazam",
"poster": "https://image.tmdb.org/t/p/w1280/xnopI5Xtky18MPhK40cZAGAOVeV.jpg",
"overview": "A boy is given the ability to become an adult superhero in times of need with a single magic word.",
"release_date": "2019-03-23"
}]'
{"updateId":0}%
❯ curl -v http://192.168.49.2:31317/indexes/movies/documents
* Trying 192.168.49.2:31317...
* Connected to 192.168.49.2 (192.168.49.2) port 31317 (#0)
> GET /indexes/movies/documents HTTP/1.1
> Host: 192.168.49.2:31317
> User-Agent: curl/7.73.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-length: 248
< content-type: application/json
< date: Tue, 12 Oct 2021 14:19:41 GMT
<
* Connection #0 to host 192.168.49.2 left intact
[{"id":287947,"title":"Shazam","poster":"https://image.tmdb.org/t/p/w1280/xnopI5Xtky18MPhK40cZAGAOVeV.jpg","overview":"A boy is given the ability to become an adult superhero in times of need with a single magic word.","release_date":"2019-03-23"}]%
取れました。
再起動してみます。
❯ kubectl rollout restart statefulset meilisearch
...
❯ kubectl get po
NAME READY STATUS RESTARTS AGE
meilisearch-0 1/1 Running 0 111s
❯ curl -v http://192.168.49.2:31317/indexes/movies/documents
* Trying 192.168.49.2:31317...
* Connected to 192.168.49.2 (192.168.49.2) port 31317 (#0)
> GET /indexes/movies/documents HTTP/1.1
> Host: 192.168.49.2:31317
> User-Agent: curl/7.73.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-length: 248
< content-type: application/json
< date: Tue, 12 Oct 2021 14:21:48 GMT
<
* Connection #0 to host 192.168.49.2 left intact
[{"id":287947,"title":"Shazam","poster":"https://image.tmdb.org/t/p/w1280/xnopI5Xtky18MPhK40cZAGAOVeV.jpg","overview":"A boy is given the ability to become an adult superhero in times of need with a single magic word.","release_date":"2019-03-23"}]%
ちゃんとデータが取得できました
冗長化してみる(PVCあり２台)
１台起動してデータを入れた状態で２台目を起動するとどうなるのかやってみます。
values.yaml
replicaCount: 2
service:
type: NodePort
persistence:
enabled: true
❯ kubectl logs -f meilisearch-1
888b d888 d8b 888 d8b .d8888b. 888
8888b d8888 Y8P 888 Y8P d88P Y88b 888
88888b.d88888 888 Y88b. 888
888Y88888P888 .d88b. 888 888 888 "Y888b. .d88b. 8888b. 888d888 .d8888b 88888b.
888 Y888P 888 d8P Y8b 888 888 888 "Y88b. d8P Y8b "88b 888P" d88P" 888 "88b
888 Y8P 888 88888888 888 888 888 "888 88888888 .d888888 888 888 888 888
888 " 888 Y8b. 888 888 888 Y88b d88P Y8b. 888 888 888 Y88b. 888 888
888 888 "Y8888 888 888 888 "Y8888P" "Y8888 "Y888888 888 "Y8888P 888 888
Database path: "./data.ms"
Server listening on: "http://0.0.0.0:7700"
Environment: "development"
Commit SHA: "3172c96042747e1ec3d27666b2bbc5ca507c56a7"
Commit date: "2021-10-11T14:27:46+00:00"
Package version: "0.23.0"
Anonymous telemetry: "Disabled"
No master key found; The server will accept unidentified requests. If you need some protection in development mode, please export a key: export MEILI_MASTER_KEY=xxx
Documentation: https://docs.meilisearch.com
Source code: https://github.com/meilisearch/meilisearch
Contact: https://docs.meilisearch.com/resources/contact.html or bonjour@meilisearch.com
[2021-10-12T14:25:00Z INFO actix_server::builder] Starting 24 workers
[2021-10-12T14:25:00Z INFO actix_server::builder] Starting "actix-web-service-0.0.0.0:7700" service on 0.0.0.0:7700
[2021-10-12T14:25:00Z ERROR meilisearch_lib::index_controller::updates::store] Fatal error while processing an update that requires the update store to shutdown: Internal error: Resource temporarily unavailable (os error 11)
[2021-10-12T14:25:00Z ERROR meilisearch_lib::index_controller::updates::store] Update store loop exited.
なんかエラーが出てますね。
[2021-10-12T14:26:20Z INFO actix_web::middleware::logger] 172.17.0.1:58946 "GET /health HTTP/1.1" 200 22 "-" "kube-probe/1.22" 0.000051
しばらくして、health checkが通って起動しました
❯ kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-26f748a1-e02c-49d0-87ef-439be1b73fa7 10Gi RWO Delete Bound default/meilisearch standard 11m
❯ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
meilisearch Bound pvc-26f748a1-e02c-49d0-87ef-439be1b73fa7 10Gi RWO standard 11m
アクセスしてみます
[2021-10-12T14:28:44Z INFO actix_web::middleware::logger] 172.17.0.1:24365 "GET /indexes/movies/documents HTTP/1.1" 500 187 "-" "curl/7.73.0" 0.000374
[2021-10-12T14:28:47Z INFO actix_web::middleware::logger] 172.17.0.1:37274 "GET /indexes/movies/documents HTTP/1.1" 500 187 "-" "curl/7.73.0" 0.000341
[2021-10-12T14:28:48Z INFO actix_web::middleware::logger] 172.17.0.1:45434 "GET /indexes/movies/documents HTTP/1.1" 500 187 "-" "curl/7.73.0" 0.000276
[2021-10-12T14:28:49Z INFO actix_web::middleware::logger] 172.17.0.1:28171 "GET /indexes/movies/documents HTTP/1.1" 500 187 "-" "curl/7.73.0" 0.000261
[2021-10-12T14:28:49Z INFO actix_web::middleware::logger] 172.17.0.1:52792 "GET /indexes/movies/documents HTTP/1.1" 500 187 "-" "curl/7.73.0" 0.000224
[2021-10-12T14:28:50Z INFO actix_web::middleware::logger] 172.17.0.1:51697 "GET /indexes/movies/documents HTTP/1.1" 500 187 "-" "curl/7.73.0" 0.000226
[2021-10-12T14:28:51Z INFO actix_web::middleware::logger] 172.17.0.1:52843 "GET /indexes/movies/documents HTTP/1.1" 500 187 "-" "curl/7.73.0" 0.000310
meilisearch-1 の方にアクセスされた場合は 500 になってしまいました。
❯ curl -v http://192.168.49.2:31317/indexes/movies/documents
* Trying 192.168.49.2:31317...
* Connected to 192.168.49.2 (192.168.49.2) port 31317 (#0)
> GET /indexes/movies/documents HTTP/1.1
> Host: 192.168.49.2:31317
> User-Agent: curl/7.73.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 500 Internal Server Error
< content-length: 187
< content-type: application/json
< date: Tue, 12 Oct 2021 14:35:09 GMT
<
* Connection #0 to host 192.168.49.2 left intact
{"message":"Internal Error: Resource temporarily unavailable (os error 11)","errorCode":"internal","errorType":"internal_error","errorLink":"https://docs.meilisearch.com/errors#internal"}%
やっぱり同じVolumeを参照しているのがまずいような？ 😣
ただIssueも無いし、使い方がおかしいのか？ ※後で調べる
MEILI_MASTER_KEY
meilisearchの MEILI_ENV を production に設定する場合 MEILI_MASTER_KEY が必要になります。
どうやらSecretが自動的に作成されるようです。
やってみます。
values.yaml
replicaCount: 2
service:
type: NodePort
persistence:
enabled: true
environment:
MEILI_ENV: production
❯ helm template meilisearch/meilisearch -f values.yaml
...
# Source: meilisearch/templates/master-key-secret.yaml
apiVersion: v1
kind: Secret
metadata:
name: RELEASE-NAME-meilisearch-master-key
labels:
app.kubernetes.io/name: meilisearch
helm.sh/chart: meilisearch-0.1.18
app.kubernetes.io/instance: RELEASE-NAME
app.kubernetes.io/managed-by: Helm
data:
MEILI_MASTER_KEY: aTNrV002UzBseVRPSzREQldHN0s=
...
たしかに追加されました。
-
meilisearch-master-keyという名前のSecretを探しにいく - あれば、その中の
MEILI_MASTER_KEYを使ってSecretを生成する - なければ、
{{ randAlphaNum 20 | b64enc }}でランダムな値を生成する
ということをやってそう。
ではSecretを事前に作っておけば、値を設定できる？
やってみます。
secret-master-key.yaml
apiVersion: v1
kind: Secret
metadata:
name: meilisearch-master-key
type: Opaque
stringData:
MEILI_MASTER_KEY: apikey
❯ kubectl apply -f secret-master-key.yaml
secret/meilisearch-master-key created
❯ kubectl describe secret meilisearch-master-key
Name: meilisearch-master-key
Namespace: default
Labels: <none>
Annotations: <none>
Type: Opaque
Data
====
MEILI_MASTER_KEY: 6 bytes
values.yaml
replicaCount: 1
service:
type: NodePort
persistence:
enabled: true
environment:
MEILI_ENV: production
> ./upgrade.sh
...
Error: UPGRADE FAILED: rendered manifests contain a resource that already exists. Unable to continue with update: Secret "meilisearch-master-key" in namespace "default" exists and cannot be imported into the current release: invalid ownership metadata; label validation error: missing key "app.kubernetes.io/managed-by": must be set to "Helm"; annotation validation error: missing key "meta.helm.sh/release-name": must be set to "meilisearch"; annotation validation error: missing key "meta.helm.sh/release-namespace": must be set to "default"
エラーになりました。
なるほど、annotationとlabelsを設定しておかないとHelmから上書きできないんですね。
直してみます。
secret-master-key.yaml
apiVersion: v1
kind: Secret
metadata:
name: meilisearch-master-key
annotations:
meta.helm.sh/release-name: meilisearch
meta.helm.sh/release-namespace: default
labels:
app.kubernetes.io/managed-by: Helm
type: Opaque
stringData:
MEILI_MASTER_KEY: apikey
❯ kubectl apply -f secret-master-key.yaml
...
❯ ./upgrade.sh
"meilisearch" already exists with the same configuration, skipping
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "meilisearch" chart repository
...Successfully got an update from the "bitnami" chart repository
Update Complete. ⎈Happy Helming!⎈
Release "meilisearch" has been upgraded. Happy Helming!
NAME: meilisearch
LAST DEPLOYED: Wed Oct 13 00:08:20 2021
NAMESPACE: default
STATUS: deployed
REVISION: 9
NOTES:
1. Get the application URL by running these commands:
export NODE_PORT=$(kubectl get --namespace default -o jsonpath="{.spec.ports[0].nodePort}" services meilisearch)
export NODE_IP=$(kubectl get nodes --namespace default -o jsonpath="{.items[0].status.addresses[0].address}")
echo http://$NODE_IP:$NODE_PORT
デプロイできました。
❯ kubectl describe secret meilisearch-master-key
Name: meilisearch-master-key
Namespace: default
Labels: app.kubernetes.io/instance=meilisearch
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=meilisearch
helm.sh/chart=meilisearch-0.1.18
Annotations: meta.helm.sh/release-name: meilisearch
meta.helm.sh/release-namespace: default
Type: Opaque
Data
====
MEILI_MASTER_KEY: 6 bytes
labelsが増えているのでhelmによって上書きされたっぽいですね。
※ただ、これやるとExternalSecretみたいなのと併用した時に相性悪いかもしれないので要調査
❯ curl -v -H "X-Meili-API-Key: apikey" http://192.168.49.2:31317/indexes/movies/documents
* Trying 192.168.49.2:31317...
* Connected to 192.168.49.2 (192.168.49.2) port 31317 (#0)
> GET /indexes/movies/documents HTTP/1.1
> Host: 192.168.49.2:31317
> User-Agent: curl/7.73.0
> Accept: */*
> X-Meili-API-Key: apikey
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-length: 248
< content-type: application/json
< date: Tue, 12 Oct 2021 15:14:25 GMT
<
* Connection #0 to host 192.168.49.2 left intact
[{"id":287947,"title":"Shazam","poster":"https://image.tmdb.org/t/p/w1280/xnopI5Xtky18MPhK40cZAGAOVeV.jpg","overview":"A boy is given the ability to become an adult superhero in times of need with a single magic word.","release_date":"2019-03-23"}]%
X-Meili-API-Key ヘッダにkeyを渡してアクセスできました。
まとめ
長くなってきたので、今回はこの辺で...
-
冗長化＆PVCでデータを永続化したい場合がまだ上手く行ってない
- 次回以降で調査するかも
- master-keyの管理方法はSecretManagerで管理してExternalSecretで取得したい
- helmによってSecretが生成されてしまうので、その辺どうするかは要検討
- そもそも諦めてenvFromのsecretRefで他のpodから
meilisearch-master-keyを参照するのがいいのか？
Discussion