Zenn
👏

箱庭にPrometheusを入れてみた（ハマった編）

2024/12/31に公開
Prometheus
tech

prometheusが起動しない

systemdに.serviceファイル作って起動してもfailedとなる

ban@UoVb:~/prometheus/prometheus-2.27.1.linux-amd64$ sudo systemctl status prometheus
● prometheus.service - Prometheus
     Loaded: loaded (/etc/systemd/system/prometheus.service; disabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Thu 2022-11-24 10:17:12 JST; 2min 52s ago
    Process: 32574 ExecStart=/usr/local/bin/prometheus (code=exited, status=2)
   Main PID: 32574 (code=exited, status=2)

11月 24 10:17:11 UoVb systemd[1]: prometheus.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
11月 24 10:17:11 UoVb systemd[1]: prometheus.service: Failed with result 'exit-code'.
11月 24 10:17:12 UoVb systemd[1]: prometheus.service: Scheduled restart job, restart counter is at 5.
11月 24 10:17:12 UoVb systemd[1]: Stopped Prometheus.
11月 24 10:17:12 UoVb systemd[1]: prometheus.service: Start request repeated too quickly.
11月 24 10:17:12 UoVb systemd[1]: prometheus.service: Failed with result 'exit-code'.
11月 24 10:17:12 UoVb systemd[1]: Failed to start Prometheus.
ban@UoVb:~/prometheus/prometheus-2.27.1.linux-amd64$

ログを見るとconfigファイルが無いと言われている模様

ban@UoVb:~/prometheus/prometheus-2.27.1.linux-amd64$ sudo cat /var/log/syslog
Nov 24 10:17:01 UoVb CRON[32548]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Nov 24 10:17:10 UoVb systemd[1]: Started Prometheus.
Nov 24 10:17:10 UoVb prometheus[32556]: level=error ts=2022-11-24T01:17:10.808Z caller=main.go:355 msg="Error loading config (--config.file=prometheus.yml)" err="open prometheus.yml: no such file or directory"
Nov 24 10:17:10 UoVb systemd[1]: prometheus.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Nov 24 10:17:10 UoVb systemd[1]: prometheus.service: Failed with result 'exit-code'.
Nov 24 10:17:11 UoVb systemd[1]: prometheus.service: Scheduled restart job, restart counter is at 1.
Nov 24 10:17:11 UoVb systemd[1]: Stopped Prometheus.
Nov 24 10:17:11 UoVb systemd[1]: Started Prometheus.
Nov 24 10:17:11 UoVb prometheus[32560]: level=error ts=2022-11-24T01:17:11.119Z caller=main.go:355 msg="Error loading config (--config.file=prometheus.yml)" err="open prometheus.yml: no such file or directory"
Nov 24 10:17:11 UoVb systemd[1]: prometheus.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Nov 24 10:17:11 UoVb systemd[1]: prometheus.service: Failed with result 'exit-code'.
Nov 24 10:17:11 UoVb systemd[1]: prometheus.service: Scheduled restart job, restart counter is at 2.
Nov 24 10:17:11 UoVb systemd[1]: Stopped Prometheus.
Nov 24 10:17:11 UoVb systemd[1]: Started Prometheus.
Nov 24 10:17:11 UoVb prometheus[32565]: level=error ts=2022-11-24T01:17:11.368Z caller=main.go:355 msg="Error loading config (--config.file=prometheus.yml)" err="open prometheus.yml: no such file or directory"
Nov 24 10:17:11 UoVb systemd[1]: prometheus.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Nov 24 10:17:11 UoVb systemd[1]: prometheus.service: Failed with result 'exit-code'.
Nov 24 10:17:11 UoVb systemd[1]: prometheus.service: Scheduled restart job, restart counter is at 3.
Nov 24 10:17:11 UoVb systemd[1]: Stopped Prometheus.
Nov 24 10:17:11 UoVb systemd[1]: Started Prometheus.
Nov 24 10:17:11 UoVb prometheus[32570]: level=error ts=2022-11-24T01:17:11.624Z caller=main.go:355 msg="Error loading config (--config.file=prometheus.yml)" err="open prometheus.yml: no such file or directory"
Nov 24 10:17:11 UoVb systemd[1]: prometheus.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Nov 24 10:17:11 UoVb systemd[1]: prometheus.service: Failed with result 'exit-code'.
Nov 24 10:17:11 UoVb systemd[1]: prometheus.service: Scheduled restart job, restart counter is at 4.
Nov 24 10:17:11 UoVb systemd[1]: Stopped Prometheus.
Nov 24 10:17:11 UoVb systemd[1]: Started Prometheus.
Nov 24 10:17:11 UoVb prometheus[32574]: level=error ts=2022-11-24T01:17:11.875Z caller=main.go:355 msg="Error loading config (--config.file=prometheus.yml)" err="open prometheus.yml: no such file or directory"
Nov 24 10:17:11 UoVb systemd[1]: prometheus.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Nov 24 10:17:11 UoVb systemd[1]: prometheus.service: Failed with result 'exit-code'.
Nov 24 10:17:12 UoVb systemd[1]: prometheus.service: Scheduled restart job, restart counter is at 5.
Nov 24 10:17:12 UoVb systemd[1]: Stopped Prometheus.
Nov 24 10:17:12 UoVb systemd[1]: prometheus.service: Start request repeated too quickly.
Nov 24 10:17:12 UoVb systemd[1]: prometheus.service: Failed with result 'exit-code'.
Nov 24 10:17:12 UoVb systemd[1]: Failed to start Prometheus.
ban@UoVb:~/prometheus/prometheus-2.27.1.linux-amd64$

configファイルの構文自体は合っている模様

ban@UoVb:~/prometheus/prometheus-2.27.1.linux-amd64$ promtool check config /etc/prometheus/prometheus.yml
Checking /etc/prometheus/prometheus.yml
  SUCCESS: 0 rule files found

ban@UoVb:~/prometheus/prometheus-2.27.1.linux-amd64$

systemdに作ったファイルもちゃんとconfigファイルを指定している

ban@UoVb:~/prometheus/prometheus-2.27.1.linux-amd64$ sudo cat /etc/systemd/system/prometheus.service
[Unit]

Description=Prometheus

Wants=network-online.target

After=network-online.target



[Service]

User=prometheus

Group=prometheus

Restart=always

Type=simple

ExecStart=/usr/local/bin/prometheus \

 --config.file=/etc/prometheus/prometheus.yml \

 --storage.tsdb.path=/var/lib/prometheus/ \

 --web.console.templates=/etc/prometheus/consoles \

 --web.console.libraries=/etc/prometheus/console_libraries \

 --web.listen-address=0.0.0.0:9090


[Install]

WantedBy=multi-user.target

ban@UoVb:~/prometheus/prometheus-2.27.1.linux-amd64$

いろいろ調べてみたけど結局よく分からず、一縷の望みをかけてprometheus.serviceを修正してみる

  • 修正前
[Service]

User=prometheus

Group=prometheus

Restart=always

Type=simple

ExecStart=/usr/local/bin/prometheus \

--config.file=/etc/prometheus/prometheus.yml \

--storage.tsdb.path=/var/lib/prometheus/ \

--web.console.templates=/etc/prometheus/consoles \

--web.console.libraries=/etc/prometheus/console_libraries \

--web.listen-address=0.0.0.0:9090
  • 修正後
[Service]

User=prometheus

Group=prometheus

Restart=always

Type=simple

ExecStart=/usr/local/bin/prometheus --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/var/lib/prometheus/ --web.console.templates=/etc/prometheus/consoles --web.console.libraries=/etc/prometheus/console_libraries --web.listen-address=0.0.0.0:9090

serviceを起動しようとしたらdaemonをreloadしなさいと怒られたのでreload実行

ban@UoVb:~/prometheus/prometheus-2.27.1.linux-amd64$ sudo systemctl start prometheus
Warning: The unit file, source configuration file or drop-ins of prometheus.service changed on disk. Run 'systemctl daemon-reload' to reload units.
ban@UoVb:~/prometheus/prometheus-2.27.1.linux-amd64$

ban@UoVb:~/prometheus/prometheus-2.27.1.linux-amd64$ sudo systemctl daemon-reload
ban@UoVb:~/prometheus/prometheus-2.27.1.linux-amd64$

で、prometheusをstart

ban@UoVb:~/prometheus/prometheus-2.27.1.linux-amd64$ sudo systemctl start prometheus
ban@UoVb:~/prometheus/prometheus-2.27.1.linux-amd64$

動いた・・・

ban@UoVb:~/prometheus/prometheus-2.27.1.linux-amd64$ sudo systemctl status prometheus
● prometheus.service - Prometheus
     Loaded: loaded (/etc/systemd/system/prometheus.service; disabled; vendor preset: enabled)
     Active: active (running) since Fri 2022-11-25 15:16:38 JST; 5s ago
   Main PID: 36941 (prometheus)
      Tasks: 6 (limit: 4618)
     Memory: 17.0M
     CGroup: /system.slice/prometheus.service
             mq36941 /usr/local/bin/prometheus --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/var/lib/prometheus/ --web.console.templates=/etc/prometheus/consoles --web.console.lib>

11月 25 15:16:38 UoVb prometheus[36941]: level=info ts=2022-11-25T06:16:38.541Z caller=head.go:755 component=tsdb msg="On-disk memory mappable chunks replay completed" duration=3.101μs
11月 25 15:16:38 UoVb prometheus[36941]: level=info ts=2022-11-25T06:16:38.541Z caller=head.go:761 component=tsdb msg="Replaying WAL, this may take a while"
11月 25 15:16:38 UoVb prometheus[36941]: level=info ts=2022-11-25T06:16:38.543Z caller=tls_config.go:191 component=web msg="TLS is disabled." http2=false
11月 25 15:16:38 UoVb prometheus[36941]: level=info ts=2022-11-25T06:16:38.543Z caller=head.go:813 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0
11月 25 15:16:38 UoVb prometheus[36941]: level=info ts=2022-11-25T06:16:38.543Z caller=head.go:818 component=tsdb msg="WAL replay completed" checkpoint_replay_duration=26.687μs wal_replay_duration=2.>
11月 25 15:16:38 UoVb prometheus[36941]: level=info ts=2022-11-25T06:16:38.545Z caller=main.go:828 fs_type=EXT4_SUPER_MAGIC
11月 25 15:16:38 UoVb prometheus[36941]: level=info ts=2022-11-25T06:16:38.545Z caller=main.go:831 msg="TSDB started"
11月 25 15:16:38 UoVb prometheus[36941]: level=info ts=2022-11-25T06:16:38.545Z caller=main.go:957 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
11月 25 15:16:38 UoVb prometheus[36941]: level=info ts=2022-11-25T06:16:38.546Z caller=main.go:988 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=>
11月 25 15:16:38 UoVb prometheus[36941]: level=info ts=2022-11-25T06:16:38.546Z caller=main.go:775 msg="Server is ready to receive web requests."
ban@UoVb:~/prometheus/prometheus-2.27.1.linux-amd64$

promtool check configとはいったい・・・

Discussion