Closed6
k8sで動くgolang apiのメトリクスをgrafana cloudで可視化する

実行するコードはこれ
grafana alloyをクラスタへインストールし、それ経由でメトリクスを転送する
クラスタはEKSで

やること
- イメージビルドしてECRへ送る
- デプロイする
- grafana alloyを入れる
- grafana cloudで確認する

deployment
こいつのメトリクスを取る
apiVersion: apps/v1
kind: Deployment
metadata:
name: golang-grafana-cloud
namespace: default
labels:
app: golang-grafana-cloud
spec:
replicas: 1
selector:
matchLabels:
app: golang-grafana-cloud
template:
metadata:
labels:
app: golang-grafana-cloud
spec:
containers:
- name: golang-grafana-cloud
image: <account id>.dkr.ecr.<region>.amazonaws.com/golang-grafana-cloud:latest
ports:
- containerPort: 8090
name: metrics
protocol: TCP
resources:
limits:
cpu: 100m
memory: 128Mi
requests:
cpu: 50m
memory: 64Mi
restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
name: golang-grafana-cloud
namespace: default
labels:
app: golang-grafana-cloud
spec:
selector:
app: golang-grafana-cloud
ports:
- name: http
port: 8090
targetPort: 8090
protocol: TCP
type: ClusterIP
動作確認
$ kubectl port-forward service/golang-grafana-cloud 8090:8090
$ curl -s http://localhost:8090/metrics | grep 'promhttp_metric_handler_requests_total{code="200"}'
promhttp_metric_handler_requests_total{code="200"} 1598

grafana alloyを入れる
helm chartで入れる
argocd app
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: alloy
namespace: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io/background
spec:
project: default
sources:
- repoURL: https://grafana.github.io/helm-charts
targetRevision: 0.10.1
chart: alloy
... 略
values.yaml
alloy:
configMap:
content: |-
logging {
level = "info"
format = "logfmt"
}
// Prometheus metrics collection
discovery.kubernetes "pods" {
role = "pod"
}
discovery.relabel "metrics" {
targets = discovery.kubernetes.pods.targets
// Keep only pods with specific labels
rule {
source_labels = ["__meta_kubernetes_pod_label_app"]
regex = "golang-grafana-cloud"
action = "keep"
}
// Set the metrics path
rule {
target_label = "__metrics_path__"
replacement = "/metrics"
}
// Set the port
rule {
source_labels = ["__meta_kubernetes_pod_container_port_number"]
regex = "8090"
action = "keep"
}
// Add custom labels
rule {
source_labels = ["__meta_kubernetes_pod_label_app"]
target_label = "app"
action = "replace"
}
rule {
source_labels = ["__meta_kubernetes_namespace"]
target_label = "namespace"
action = "replace"
}
}
prometheus.scrape "golang_metrics" {
targets = discovery.relabel.metrics.output
forward_to = [prometheus.remote_write.grafanacloud.receiver]
scrape_interval = "15s"
}
prometheus.remote_write "grafanacloud" {
endpoint {
url = "<grafana cloud prometheus url>"
basic_auth {
username = "<username>"
password = "<password>"
}
}
}

memo
grafana cloudへログインし、prometheus(mimir)の詳細画面から接続情報を確認
alloyのvalues.yamlへ書き込む
パスワードはAPI tokenを新規発行して入力

できた
このスクラップは1ヶ月前にクローズされました