Closed6

k8sで動くgolang apiのメトリクスをgrafana cloudで可視化する

not75743not75743

やること

  • イメージビルドしてECRへ送る
  • デプロイする
  • grafana alloyを入れる
  • grafana cloudで確認する
not75743not75743

deployment

こいつのメトリクスを取る

apiVersion: apps/v1
kind: Deployment
metadata:
  name: golang-grafana-cloud
  namespace: default
  labels:
    app: golang-grafana-cloud
spec:
  replicas: 1
  selector:
    matchLabels:
      app: golang-grafana-cloud
  template:
    metadata:
      labels:
        app: golang-grafana-cloud
    spec:
      containers:
      - name: golang-grafana-cloud
        image: <account id>.dkr.ecr.<region>.amazonaws.com/golang-grafana-cloud:latest
        ports:
        - containerPort: 8090
          name: metrics
          protocol: TCP
        resources:
          limits:
            cpu: 100m
            memory: 128Mi
          requests:
            cpu: 50m
            memory: 64Mi
      restartPolicy: Always
---

apiVersion: v1
kind: Service
metadata:
  name: golang-grafana-cloud
  namespace: default
  labels:
    app: golang-grafana-cloud
spec:
  selector:
    app: golang-grafana-cloud
  ports:
  - name: http
    port: 8090
    targetPort: 8090
    protocol: TCP
  type: ClusterIP

動作確認

$ kubectl port-forward service/golang-grafana-cloud 8090:8090
$ curl -s http://localhost:8090/metrics | grep 'promhttp_metric_handler_requests_total{code="200"}'
promhttp_metric_handler_requests_total{code="200"} 1598
not75743not75743

grafana alloyを入れる

helm chartで入れる
argocd app

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: alloy
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io/background
spec:
  project: default
  sources:
    - repoURL: https://grafana.github.io/helm-charts
      targetRevision: 0.10.1
      chart: alloy
...

values.yaml

alloy:
  configMap:
    content: |-
      logging {
        level = "info"
        format = "logfmt"
      }
      
      // Prometheus metrics collection
      discovery.kubernetes "pods" {
        role = "pod"
      }
      
      discovery.relabel "metrics" {
        targets = discovery.kubernetes.pods.targets
        
        // Keep only pods with specific labels
        rule {
          source_labels = ["__meta_kubernetes_pod_label_app"]
          regex         = "golang-grafana-cloud"
          action        = "keep"
        }
        
        // Set the metrics path
        rule {
          target_label = "__metrics_path__"
          replacement  = "/metrics"
        }
        
        // Set the port
        rule {
          source_labels = ["__meta_kubernetes_pod_container_port_number"]
          regex         = "8090"
          action        = "keep"
        }
        
        // Add custom labels
        rule {
          source_labels = ["__meta_kubernetes_pod_label_app"]
          target_label  = "app"
          action        = "replace"
        }
        
        rule {
          source_labels = ["__meta_kubernetes_namespace"]
          target_label  = "namespace"
          action        = "replace"
        }
      }
      
      prometheus.scrape "golang_metrics" {
        targets    = discovery.relabel.metrics.output
        forward_to = [prometheus.remote_write.grafanacloud.receiver]
        scrape_interval = "15s"
      }
      
      prometheus.remote_write "grafanacloud" {
        endpoint {
          url = "<grafana cloud prometheus url>"

          basic_auth {
            username = "<username>"
            password = "<password>"
          }
        }
      }   
not75743not75743

memo

grafana cloudへログインし、prometheus(mimir)の詳細画面から接続情報を確認
alloyのvalues.yamlへ書き込む

パスワードはAPI tokenを新規発行して入力

このスクラップは1ヶ月前にクローズされました