Building OpenMetadata on Local Kubernetres
自宅のKubernetes環境にOpenMetadataを構築し、Cloudflare Tunnelで外部公開して遊んでみたので備忘録です。
ローカルKubernetesにOpenMetadataをデプロイする
手順については、次の2つの公式ドキュメントを参照し、一部修正を行い構築しました。
ローカルマシンスペックについて
公式ドキュメントには、4vCPUと8GiBのメモリを備えた minikube クラスターをスペック条件と記載があります。
minikube start --cpus=4 --memory=8192
最低でもこのスペックは確保しておきます。
今回はminikubeではなく、kubeadmで構築したクラスターで各ノードは4coreのRAM16GiBで構築しています。
事前準備
Helmをインストールする
すでにHelmをインストールされている場合はこの手順はスキップしてOKです。
今回の構成では全ノードでHelmを利用するので、各ノードで実行します。
$ curl -O https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
$ bash ./get-helm-3
$ helm version
version.BuildInfo{Version:"v3.16.4", GitCommit:"7877b45b63f95635153b29a42c0c2f4273ec45ca", GitTreeState:"clean", GoVersion:"go1.22.7"}
StorageClass を使用した動的プロビジョニング
今回Helmを使ったOpenMetadata構築を行います。
OpenMetadata HelmチャートはAirflowに依存しており、AirflowはReadWriteMany(ボリュームは多くのノードによって読み取り/書き込みとしてマウントできます)をサポートする永続ディスクを想定しています。
そのため、nfs-shareを作成し、それを永続的なストレージとして使用してます。
StorageClassを使用してPersistentVolumeを動的にプロビジョニングするには、NFSプロビジョナーをインストールする必要があります。
今回は、公式推奨のnfs-subdir-external-provisioner Helmチャートを使用します。
ただし、すでに動的ストレージが存在する場合はこの手順はスキップしてOKです。
NFSサーバの構築
私の環境にNFSサーバが存在しないので、作っていきます。
今回はワーカーノードにNFSサーバを用意しています。
$ sudo apt update
$ sudo apt install nfs-kernel-server -y
$ sudo mkdir -p /airflow
$ sudo chmod 777 /airflow
$ sudo mkdir -p /airflow/airflow-dags /airflow/airflow-logs
$ sudo chown -R nobody:nogroup /airflow
$ sudo chmod -R 777 /airflow
/etc/exportsにマウント時に指定するパスを設定
この後Airflow用のPersistentVolumeClaimを作成しますが、そのマウントパスが/airflow-dagsや/airflow-logsとするので、ルートのマウントパスは/airflowとし、その配下にそれぞれのPersistentVolumeClaimのパスが来るようにしています。
$ sudo cat /etc/exports
# /etc/exports: the access control list for filesystems which may be exported
# to NFS clients. See exports(5).
#
# Example for NFSv2 and NFSv3:
# /srv/homes hostname1(rw,sync,no_subtree_check) hostname2(ro,sync,no_subtree_check)
#
# Example for NFSv4:
# /srv/nfs4 gss/krb5i(rw,sync,fsid=0,crossmnt,no_subtree_check)
# /srv/nfs4/homes gss/krb5i(rw,sync,no_subtree_check)
#
/airflow *(rw,sync,no_root_squash,no_subtree_check) #追記
設定の反映を行う
$ sudo exportfs -ra
nfs-subdir-external-provisionerのインストール
パラメータ説明
パラメータ | 概要 |
---|---|
nfs.server | NFSサーバのホスト名もしくはIPアドレスを指定 |
nfs.path | NFSサーバ側で/etc/exportsに設定したマウントパスを指定 |
$ sudo helm install nfs-subdir-external-provisioner nfs-subdir-external-provisioner/nfs-subdir-external-provisioner --create-namespace --namespace nfs-provisioner --set nfs.server=k8s-worker01 --set nfs.path=/airflow
nfs-serverを有効にする
$ sudo systemctl enable nfs-server
$ sudo systemctl start nfs-server
$ sudo exportfs -v
/airflow <world>(sync,wdelay,hide,no_subtree_check,sec=sys,rw,secure,no_root_squash,no_all_squash)
PersistentVolumeClaimを作成
動的プロビジョニングでAirflowのdagとlogsのPersistentVolumeClaimを作成します。
openmetadata-dependencies-dags定義ファイル
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
namespace: default
name: openmetadata-dependencies-dags
labels:
storage.k8s.io/name: nfs
app: airflow
app.kubernetes.io/managed-by: Helm
annotations:
meta.helm.sh/release-name: openmetadata-dependencies
meta.helm.sh/release-namespace: default
spec:
accessModes:
- ReadWriteMany
storageClassName: nfs-client
resources:
requests:
storage: 1Gi
openmetadata-dependencies-logs定義ファイル
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
namespace: default
name: openmetadata-dependencies-logs
labels:
storage.k8s.io/name: nfs
app: airflow
app.kubernetes.io/managed-by: Helm
annotations:
meta.helm.sh/release-name: openmetadata-dependencies
meta.helm.sh/release-namespace: default
spec:
accessModes:
- ReadWriteMany
storageClassName: nfs-client
resources:
requests:
storage: 1Gi
作成した定義ファイルからPVCを作成する。
$ sudo kubectl apply -f dags_pvc.yml
persistentvolumeclaim/openmetadata-dependencies-dags created
$ sudo kubectl apply -f logs_pvc.yml
persistentvolumeclaim/openmetadata-dependencies-logs created
$ sudo kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE
openmetadata-dependencies-dags Bound pvc-82f91642-b6ce-4a97-930c-************ 1Gi RWX nfs-client <unset> 46s
openmetadata-dependencies-logs Bound pvc-d0c781f9-3784-41ef-a503-************ 1Gi RWX nfs-client <unset> 18s
ディスクの所有者と権限を手動で変更する
Airflowポッドは非ルートユーザーとして実行されるため、NFSサーバー ボリュームへの書き込みアクセス権がありません。
ここで権限を修正するには、永続ボリュームが接続されたポッドを起動し、それを1回実行します。
定義ファイル
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: null
labels:
run: my-permission-pod
name: my-permission-pod
spec:
containers:
- image: busybox
name: my-permission-pod
volumeMounts:
- name: airflow-dags
mountPath: /airflow-dags
- name: airflow-logs
mountPath: /airflow-logs
command:
- "chown -R 50000 /airflow-dags /airflow-logs"
# if needed
- "chmod -R a+rwx /airflow-dags"
volumes:
- name: airflow-logs
persistentVolumeClaim:
claimName: openmetadata-dependencies-logs
- name: airflow-dags
persistentVolumeClaim:
claimName: openmetadata-dependencies-dags
dnsPolicy: ClusterFirst
restartPolicy: Always
権限を修正するためにmy-permission-podを起動します。
$ sudo kubectl create -f permissions_pod.yml
Podの起動以外にNFSサーバにログインして以下のコマンドを実行するでもOKです。
$ sudo chown -R 50000 /airflow/airflow-dags /airflow/airflow-logs
$ sudo chmod -R a+rwx /airflow/airflow-dags
OpenMetadata依存関係値を作成する
openmetadata依存関係のAirflow Helm値をオーバーライドして、DAGとログのnfs永続ボリュームをバインドします。
values-dependencies.yml定義ファイル
airflow:
enabled: true
airflow:
image:
repository: docker.getcollate.io/openmetadata/ingestion
tag: 1.6.1
pullPolicy: "IfNotPresent"
executor: "KubernetesExecutor"
config:
# This is required for OpenMetadata UI to fetch status of DAGs
AIRFLOW__API__AUTH_BACKENDS: "airflow.api.auth.backend.session,airflow.api.auth.backend.basic_auth"
# OpenMetadata Airflow Apis Plugin DAGs Configuration
AIRFLOW__OPENMETADATA_AIRFLOW_APIS__DAG_GENERATED_CONFIGS: "/airflow-dags/dags"
# OpenMetadata Airflow Secrets Manager Configuration
AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AWS_REGION: ""
AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AWS_ACCESS_KEY_ID: ""
AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AWS_ACCESS_KEY: ""
users:
- username: admin
password: admin
role: Admin
email: spiderman@superhero.org
firstName: Peter
lastName: Parker
web:
extraVolumes:
- name: nfs-airflow-logs
persistentVolumeClaim:
claimName: openmetadata-dependencies-logs
- name: nfs-airflow-dags
persistentVolumeClaim:
claimName: openmetadata-dependencies-dags
extraVolumeMounts:
- mountPath: /airflow-logs
name: nfs-airflow-logs
- mountPath: /airflow-dags/dags
name: nfs-airflow-dags
readinessProbe:
enabled: true
initialDelaySeconds: 60
periodSeconds: 30
timeoutSeconds: 10
failureThreshold: 10
livenessProbe:
enabled: true
initialDelaySeconds: 60
periodSeconds: 30
timeoutSeconds: 10
failureThreshold: 10
postgresql:
enabled: false
workers:
enabled: false
flower:
enabled: false
redis:
enabled: false
externalDatabase:
type: mysql
host: mysql
port: 3306
database: airflow_db
user: airflow_user
passwordSecret: airflow-mysql-secrets
passwordSecretKey: airflow-mysql-password
serviceAccount:
create: true
name: "airflow"
scheduler:
logCleanup:
enabled: false
dags:
path: /airflow-dags/dags
persistence:
enabled: false
logs:
path: /airflow-logs
persistence:
enabled: false
Opensearch/MySQL用のローカルストレージを用意
ワーカーノードにてOpensearch/MySQL用のローカルストレージ領域を確保します。
$ sudo mkdir -p /mnt/data/mysql
$ sudo chmod 777 /mnt/data/mysql
$ sudo mkdir -p /mnt/data/opensearch
$ sudo chmod 777 /mnt/data/opensearch
PV/PVCの定義ファイル
apiVersion: v1
kind: PersistentVolume
metadata:
name: mysql-pv
spec:
capacity:
storage: 50Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
storageClassName: mysql-local-storage
local:
path: /mnt/data/mysql
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- k8s-worker01
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: mysql-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 50Gi
storageClassName: mysql-local-storage
kind: PersistentVolume
metadata:
name: opensearch-pv
spec:
capacity:
storage: 30Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
storageClassName: opensearch-local-storage
local:
path: /mnt/data/opensearch
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- k8s-worker01
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: opensearch-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 30Gi
storageClassName: opensearch-local-storage
$ sudo kubectl apply -f mysql-pv.yaml
$ sudo kubectl apply -f mysql-pvc.yaml
$ sudo kubectl apply -f opensearch-pv.yaml
$ sudo kubectl apply -f opensearch-pvc.yaml
$ sudo kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS VOLUMEATTRIBUTESCLASS REASON AGE
mysql-pv 50Gi RWO Retain Bound default/mysql-pvc mysql-local-storage <unset> 24s
opensearch-pv 30Gi RWO Retain Bound default/opensearch-pvc opensearch-local-storage <unset> 16s
$ sudo kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE
mysql-pvc Bound mysql-pv 50Gi RWO mysql-local-storage <unset> 25s
opensearch-pvc Bound opensearch-pv 30Gi RWO opensearch-local-storage <unset> 17s
openmetadata-dependencies-dags Bound pvc-82f91642-b6ce-4a97-930c-************ 1Gi RWX nfs-client <unset> 46s
openmetadata-dependencies-logs Bound pvc-d0c781f9-3784-41ef-a503-************ 1Gi RWX nfs-client <unset> 18s
Helm Chartsに必要なKubernetes Secretを作成する
$ sudo kubectl create secret generic mysql-secrets --from-literal=openmetadata-mysql-password=openmetadata_password
$ sudo kubectl create secret generic airflow-secrets --from-literal=openmetadata-airflow-password=admin
$ sudo kubectl create secret generic airflow-mysql-secrets --from-literal=airflow-mysql-password=airflow_pass
ローカル展開用のHelmリポジトリを追加する
$ sudo helm repo add open-metadata https://helm.open-metadata.org/
Helmリポジトリが追加されたことを確認
$ sudo helm repo list
NAME URL
open-metadata https://helm.open-metadata.org/
OpenMetadata Dependencies Helm Chart をインストールする
前提条件で作成したAirflowの
$ sudo helm install openmetadata-dependencies open-metadata/openmetadata-dependencies --values ./values-dependencies.yml
NAME: openmetadata-dependencies
LAST DEPLOYED: Tue Dec 24 12:51:11 2024
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
Pod起動確認
$ sudo kubectl get pods
OpenMetadata Helm Chartをインストールする
$ sudo helm install openmetadata open-metadata/openmetadata
公式のHelmだと、OpenMetadataのサインイン画面上に「CREATE ACCOUNT」のボタンがあり、誰でもユーザを作成できてしまいます。
それを回避するために、Helmの「openmetadata.config.authentication.enableSelfSignup」をflaseに変更する定義ファイルを作成し、Helmインストール時に定義ファイルを指定します。
openmetadata_values.ymlの定義ファイル
# Default values for OpenMetadata.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.
replicaCount: 1
# Overrides the openmetadata config file with the help of Environment Variables
# Below are defaults as per openmetadata-dependencies Helm Chart Values
openmetadata:
config:
upgradeMigrationConfigs:
debug: false
# You can pass the additional argument flags to the openmetadata-ops.sh migrate command
# Example if you want to force migration runs, use additionalArgs: "--force"
additionalArgs: ""
deployPipelinesConfig:
debug: false
additionalArgs: ""
reindexConfig:
debug: false
# You can pass the additional argument flags to the openmetadata-ops.sh reindex command
additionalArgs: ""
# Values can be OFF, ERROR, WARN, INFO, DEBUG, TRACE, or ALL
logLevel: INFO
clusterName: openmetadata
openmetadata:
host: "0.0.0.0"
# URI to use with OpenMetadata Alerts Integrations
uri: "http://openmetadata:8585"
port: 8585
adminPort: 8586
elasticsearch:
enabled: true
host: opensearch
searchType: opensearch
port: 9200
scheme: http
clusterAlias: ""
# Value in Bytes
payLoadSize: 10485760
connectionTimeoutSecs: 5
socketTimeoutSecs: 60
batchSize: 100
searchIndexMappingLanguage: "EN"
keepAliveTimeoutSecs: 600
trustStore:
enabled: false
path: ""
password:
secretRef: "elasticsearch-truststore-secrets"
secretKey: "openmetadata-elasticsearch-truststore-password"
auth:
enabled: false
username: "elasticsearch"
password:
secretRef: elasticsearch-secrets
secretKey: openmetadata-elasticsearch-password
database:
enabled: true
host: mysql
port: 3306
driverClass: com.mysql.cj.jdbc.Driver
dbScheme: mysql
databaseName: openmetadata_db
auth:
username: openmetadata_user
password:
secretRef: mysql-secrets
secretKey: openmetadata-mysql-password
dbParams: "allowPublicKeyRetrieval=true&useSSL=false&serverTimezone=UTC"
maxSize: 50
minSize: 10
initialSize: 10
checkConnectionWhileIdle: true
checkConnectionOnBorrow: true
evictionInterval: 5 minutes
minIdleTime: 1 minute
pipelineServiceClientConfig:
enabled: true
className: "org.openmetadata.service.clients.pipeline.airflow.AirflowRESTClient"
# endpoint url for airflow
apiEndpoint: http://openmetadata-dependencies-web:8080
# this will be the api endpoint url of OpenMetadata Server
metadataApiEndpoint: http://openmetadata:8585/api
# possible values are "no-ssl", "ignore", "validate"
verifySsl: "no-ssl"
hostIp: ""
ingestionIpInfoEnabled: false
# healthCheckInterval in seconds
healthCheckInterval: 300
# local path in Airflow Pod
sslCertificatePath: "/no/path"
auth:
enabled: true
username: admin
password:
secretRef: airflow-secrets
secretKey: openmetadata-airflow-password
trustStorePath: ""
trustStorePassword:
secretRef: ""
secretKey: ""
authorizer:
enabled: true
className: "org.openmetadata.service.security.DefaultAuthorizer"
containerRequestFilter: "org.openmetadata.service.security.JwtFilter"
initialAdmins:
- "admin"
allowedEmailRegistrationDomains:
- "all"
principalDomain: "open-metadata.org"
enforcePrincipalDomain: false
enableSecureSocketConnection: false
useRolesFromProvider: false
authentication:
enabled: true
clientType: public
provider: "basic"
publicKeys:
- "http://openmetadata:8585/api/v1/system/config/jwks"
authority: "https://accounts.google.com"
clientId: ""
callbackUrl: ""
responseType: id_token
jwtPrincipalClaims:
- "email"
- "preferred_username"
- "sub"
jwtPrincipalClaimsMapping: []
# jwtPrincipalClaimsMapping:
# - username:sub
# - email:email
enableSelfSignup: false
oidcConfiguration:
enabled: false
oidcType: ""
clientId:
secretRef: oidc-secrets
secretKey: openmetadata-oidc-client-id
clientSecret:
secretRef: oidc-secrets
secretKey: openmetadata-oidc-client-secret
scope: "openid email profile"
discoveryUri: ""
useNonce: true
preferredJwsAlgorithm: RS256
responseType: code
disablePkce: true
callbackUrl: http://openmetadata:8585/callback
serverUrl: http://openmetadata:8585
clientAuthenticationMethod: client_secret_post
tenant: ""
maxClockSkew: ""
tokenValidity: "3600"
customParams: ""
ldapConfiguration:
host: localhost
port: 10636
dnAdminPrincipal: "cn=admin,dc=example,dc=com"
dnAdminPassword:
secretRef: ldap-admin-secret
secretKey: openmetadata-ldap-secret
userBaseDN: "ou=people,dc=example,dc=com"
mailAttributeName: email
maxPoolSize: 3
sslEnabled: false
groupBaseDN: ""
roleAdminName: ""
allAttributeName: ""
usernameAttributeName: ""
groupAttributeName: ""
groupAttributeValue: ""
groupMemberAttributeName: ""
authRolesMapping: ""
authReassignRoles: []
# Possible values are CustomTrustStore, HostName, JVMDefault, TrustAll
truststoreConfigType: TrustAll
trustStoreConfig:
customTrustManagerConfig:
trustStoreFilePath: ""
trustStoreFilePassword:
secretRef: ""
secretKey: ""
trustStoreFileFormat: ""
verifyHostname: true
examineValidityDates: true
hostNameConfig:
allowWildCards: false
acceptableHostNames: []
jvmDefaultConfig:
verifyHostname: true
trustAllConfig:
examineValidityDates: true
saml:
debugMode: false
idp:
entityId: ""
ssoLoginUrl: ""
idpX509Certificate:
secretRef: ""
secretKey: ""
authorityUrl: "http://openmetadata:8585/api/v1/saml/login"
nameId: "urn:oasis:names:tc:SAML:2.0:nameid-format:emailAddress"
sp:
entityId: "http://openmetadata:8585/api/v1/saml/metadata"
acs: "http://openmetadata:8585/api/v1/saml/acs"
spX509Certificate:
secretRef: ""
secretKey: ""
spPrivateKey:
secretRef: ""
secretKey: ""
callback: "http://openmetadata:8585/saml/callback"
security:
strictMode: false
validateXml: false
tokenValidity: 3600
sendEncryptedNameId: false
sendSignedAuthRequest: false
signSpMetadata: false
wantMessagesSigned: false
wantAssertionsSigned: false
wantAssertionEncrypted: false
keyStoreFilePath: ""
keyStoreAlias:
secretRef: ""
secretKey: ""
keyStorePassword:
secretRef: ""
secretKey: ""
jwtTokenConfiguration:
enabled: true
# File Path on Airflow Container
rsapublicKeyFilePath: "./conf/public_key.der"
# File Path on Airflow Container
rsaprivateKeyFilePath: "./conf/private_key.der"
jwtissuer: "open-metadata.org"
keyId: "Gb389a-9f76-gdjs-a92j-0242bk94356"
fernetkey:
value: "jJ/9sz0g0OHxsfxOoSfdFdmk3ysNmPRnH3TUAbz3IHA="
secretRef: ""
secretKey: ""
eventMonitor:
enabled: true
# Possible values are prometheus and cloudwatch
type: prometheus
batchSize: 10
pathPattern:
- "/api/v1/tables/*"
- "/api/v1/health-check"
# For value p99=0.99, p90=0.90, p50=0.50 etc.
latency: []
# - "p99=0.99"
# - "p90=0.90"
# - "p50=0.50"
smtpConfig:
enableSmtpServer: false
emailingEntity: "OpenMetadata"
supportUrl: "https://slack.open-metadata.org"
transportationStrategy: "SMTP_TLS"
openMetadataUrl: "http://openmetadata:8585"
serverEndpoint: ""
serverPort: ""
senderMail: ""
username: ""
password:
secretRef: ""
secretKey: ""
secretsManager:
enabled: true
# Possible values are db, aws, aws-ssm, managed-aws, managed-aws-ssm, in-memory, managed-azure-kv, azure-kv, gcp
provider: db
# Define the secret key ID as /<prefix>/<clusterName>/<key> for AWS
# Define the secret key ID as <prefix>-<clusterName>-<key> for Azure
prefix: ""
# Add tags to the created resource, e.g., in AWS. Format is `[key1:value1,key2:value2,...]`
tags: []
additionalParameters:
enabled: false
region: ""
# For AWS
accessKeyId:
secretRef: ""
secretKey: ""
secretAccessKey:
secretRef: ""
secretKey: ""
# accessKeyId:
# secretRef: aws-access-key-secret
# secretKey: aws-key-secret
# secretAccessKey:
# secretRef: aws-secret-access-key-secret
# secretKey: aws-key-secret
# For Azure
clientId:
secretRef: ""
secretKey: ""
clientSecret:
secretRef: ""
secretKey: ""
tenantId:
secretRef: ""
secretKey: ""
vaultName:
secretRef: ""
secretKey: ""
# clientId:
# secretRef: azure-client-id-secret
# secretKey: azure-key-secret
# clientSecret:
# secretRef: azure-client-secret
# secretKey: azure-key-secret
# tenantId:
# secretRef: azure-tenant-id-secret
# secretKey: azure-key-secret
# vaultName:
# secretRef: azure-vault-name-secret
# secretKey: azure-key-secret
# For GCP
projectId:
secretRef: ""
secretKey: ""
# projectId:
# secretRef: gcp-project-id-secret
# secretKey: gcp-key-secret
# You can create Kubernetes secrets from AWS Credentials with the below command
# kubectl create secret generic aws-key-secret \
# --from-literal=aws-access-key-secret=<access_key_id_value> \
# --from-literal=aws-secret-access-key-secret=<access_key_secret_value>
web:
enabled: true
uriPath: "/api"
hsts:
enabled: false
maxAge: "365 days"
includeSubDomains: "true"
preload: "true"
frameOptions:
enabled: false
option: "SAMEORIGIN"
origin: ""
contentTypeOptions:
enabled: false
xssProtection:
enabled: false
onXss: true
block: true
csp:
enabled: false
policy: "default-src 'self'"
reportOnlyPolicy: ""
referrerPolicy:
enabled: false
option: "SAME_ORIGIN"
permissionPolicy:
enabled: false
option: ""
cacheControl: ""
pragma: ""
networkPolicy:
# If networkPolicy is true, following values can be set
# for ingress on port 8585 and 8586
enabled: false
# Example Google SSO Auth Config
# authorizer:
# className: "org.openmetadata.service.security.DefaultAuthorizer"
# containerRequestFilter: "org.openmetadata.service.security.JwtFilter"
# initialAdmins:
# - "suresh"
# principalDomain: "open-metadata.org"
# authentication:
# provider: "google"
# publicKeys:
# - "https://www.googleapis.com/oauth2/v3/certs"
# authority: "https://accounts.google.com"
# clientId: "<client_id>"
# callbackUrl: "<callback_url>"
image:
repository: docker.getcollate.io/openmetadata/server
# Overrides the image tag whose default is the chart appVersion.
tag: ""
pullPolicy: "Always"
sidecars: []
# - name: "busybox"
# image: "busybox:1.34.1"
# imagePullPolicy: "Always"
# command: ["ls"]
# args: ["-latr", "/usr/share"]
# env:
# - name: DEMO
# value: "DEMO"
# volumeMounts:
# - name: extras
# mountPath: /usr/share/extras
# readOnly: true
imagePullSecrets: []
nameOverride: ""
fullnameOverride: "openmetadata"
serviceAccount:
# Specifies whether a service account should be created
create: true
# Annotations to add to the service account
annotations: {}
# The name of the service account to use.
# If not set and create is true, a name is generated using the fullname template
name: ""
automountServiceAccountToken: true
podSecurityContext: {}
# fsGroup: 2000
securityContext: {}
# capabilities:
# drop:
# - ALL
# readOnlyRootFilesystem: true
# runAsNonRoot: true
# runAsUser: 100
service:
type: ClusterIP
port: 8585
adminPort: 8586
annotations: {}
# Service monitor for Prometheus metrics
serviceMonitor:
enabled: false
interval: 30s
annotations: {}
labels: {}
ingress:
enabled: false
className: ""
annotations: {}
# kubernetes.io/tls-acme: "true"
hosts:
- host: open-metadata.local
paths:
- path: /
pathType: ImplementationSpecific
tls: []
# - secretName: tls-open-metadata.local
# hosts:
# - open-metadata.local
extraEnvs: []
# - name: MY_ENVIRONMENT_VAR
# value: the_value_goes_here
envFrom: []
# - secretRef:
# name: secret_containing_config
extraVolumes: []
# - name: extras
# emptyDir: {}
extraVolumeMounts: []
# - name: extras
# mountPath: /usr/share/extras
# readOnly: true
# Provision for InitContainers to be running after the `run-db-migration` InitContainer
extraInitContainers: []
# Provision for InitContainers to be running before the `run-db-migration` InitContainer
preMigrateInitContainers: []
resources: {}
# We usually recommend not to specify default resources and to leave this as a conscious
# choice for the user. This also increases chances charts run on environments with little
# resources, such as Minikube. If you do want to specify resources, uncomment the following
# lines, adjust them as necessary, and remove the curly braces after 'resources:'.
# limits:
# cpu: 1
# memory: 2048Mi
# requests:
# cpu: 500m
# memory: 256Mi
nodeSelector: {}
tolerations: []
affinity: {}
livenessProbe:
initialDelaySeconds: 60
periodSeconds: 30
failureThreshold: 5
httpGet:
path: /healthcheck
port: http-admin
readinessProbe:
initialDelaySeconds: 60
periodSeconds: 30
failureThreshold: 5
httpGet:
path: /
port: http
startupProbe:
periodSeconds: 60
failureThreshold: 5
successThreshold: 1
httpGet:
path: /healthcheck
port: http-admin
podDisruptionBudget:
enabled: false
config:
maxUnavailable: "1"
minAvailable: "1"
commonLabels: {}
deploymentAnnotations: {}
podAnnotations: {}
$ sudo helm install openmetadata open-metadata/openmetadata --values ./openmetadata_values.yml
Pod起動確認
$ sudo kubectl get pods
Cloudclare Tunnelを使った外部公開
Cloudclare Tunnel設定は以下の記事を参考ください。
cloudflared.yamlの定義ファイル
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: cloudflared
spec:
selector:
matchLabels:
app: cloudflared
replicas: 2 # You could also consider elastic scaling for this deployment
template:
metadata:
labels:
app: cloudflared
spec:
containers:
- name: cloudflared
image: cloudflare/cloudflared:2022.3.0
args:
- tunnel
# Points cloudflared to the config file, which configures what
# cloudflared will actually do. This file is created by a ConfigMap
# below.
- --config
- /etc/cloudflared/config/config.yaml
- run
livenessProbe:
httpGet:
# Cloudflared has a /ready endpoint which returns 200 if and only if
# it has an active connection to the edge.
path: /ready
port: 2000
failureThreshold: 1
initialDelaySeconds: 10
periodSeconds: 10
volumeMounts:
- name: config
mountPath: /etc/cloudflared/config
readOnly: true
# Each tunnel has an associated "credentials file" which authorizes machines
# to run the tunnel. cloudflared will read this file from its local filesystem,
# and it'll be stored in a k8s secret.
- name: creds
mountPath: /etc/cloudflared/creds
readOnly: true
volumes:
- name: creds
secret:
# By default, the credentials file will be created under ~/.cloudflared/<tunnel ID>.json
# when you run `cloudflared tunnel create`. You can move it into a secret by using:
# ```sh
# kubectl create secret generic tunnel-credentials \
# --from-file=credentials.json=/Users/yourusername/.cloudflared/<tunnel ID>.json
# ```
secretName: openmetadata-tunnel-credentials
# Create a config.yaml file from the ConfigMap below.
- name: config
configMap:
name: cloudflared
items:
- key: config.yaml
path: config.yaml
---
# This ConfigMap is just a way to define the cloudflared config.yaml file in k8s.
# It's useful to define it in k8s, rather than as a stand-alone .yaml file, because
# this lets you use various k8s templating solutions (e.g. Helm charts) to
# parameterize your config, instead of just using string literals.
apiVersion: v1
kind: ConfigMap
metadata:
name: cloudflared
data:
config.yaml: |
# Name of the tunnel you want to run
tunnel: openmetadata-tunnel
credentials-file: /etc/cloudflared/creds/credentials.json
# Serves the metrics server under /metrics and the readiness server under /ready
metrics: 0.0.0.0:2000
no-autoupdate: true
ingress:
- hostname: omd.kyamisama.com
service: http://openmetadata:8585
- hostname: omd.kyamisama.com
service: hello_world
- service: http_status:404
$ sudo kubectl apply -f cloudflared.yaml
ブラウザからアクセスしてみる
おわり
私自身そこまでKuberneteに詳しくないのですが、それでも公式ドキュメントやググりながらなんとか構築できたので、もし興味がある人は是非お試しください。
あといつの間にか機能が増えていたので別記事で色々と検証してみたいと思います!
その前に次回はOpenMetadataの認証認可周りをkeycloakでやる記事を書こうと思います!
Discussion