GrafanaでUbuntu Serverの監視体制をつくる（Loki・Promtail編）

ここまでのあらすじ

PrometheusとPromtailとLokiを入れてGrafanaで見れるようにした
- https://zenn.dev/alice37/scraps/e4f73486f531cf
Lokiが何のログファイルも検知しなくなった
原因がわからんのでPromtailとLokiをpurgeした

記載上はapt installじゃなくてapt-get installだけど差異ある気はしない
一応記載通りapt-getで入れる

sudo apt-get install loki promtail
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following package was automatically installed and is no longer required:
  libllvm14
Use 'sudo apt autoremove' to remove it.
The following NEW packages will be installed:
  loki promtail
0 upgraded, 2 newly installed, 0 to remove and 5 not upgraded.
Need to get 51.2 MB of archives.
After this operation, 169 MB of additional disk space will be used.
Get:1 https://apt.grafana.com stable/main amd64 loki amd64 3.0.0 [22.6 MB]
Get:2 https://apt.grafana.com stable/main amd64 promtail amd64 3.0.0 [28.6 MB]
Fetched 51.2 MB in 4s (11.7 MB/s)
Selecting previously unselected package loki.
(Reading database ... 159932 files and directories currently installed.)
Preparing to unpack .../archives/loki_3.0.0_amd64.deb ...
Unpacking loki (3.0.0) ...
Selecting previously unselected package promtail.
Preparing to unpack .../promtail_3.0.0_amd64.deb ...
Unpacking promtail (3.0.0) ...
Setting up loki (3.0.0) ...
 Post Install of a clean install
 Reload the service unit from disk
 Unmask the service
 Set the preset flag for the service unit
 Set the enabled flag for the service unit
Setting up promtail (3.0.0) ...
 Post Install of a clean install
 Reload the service unit from disk
 Unmask the service
 Set the preset flag for the service unit
 Set the enabled flag for the service unit
Scanning processes...
Scanning processor microcode...
Scanning linux images...

勝手に

systemctl status loki
● loki.service - Loki service
     Loaded: loaded (/etc/systemd/system/loki.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2024-04-22 22:14:36 JST; 1min 46s ago
   Main PID: 895850 (loki)
      Tasks: 10 (limit: 9197)
     Memory: 35.1M
        CPU: 719ms
     CGroup: /system.slice/loki.service
             └─895850 /usr/bin/loki -config.file /etc/loki/config.yml

やってくれるらしい

systemctl status promtail
● promtail.service - Promtail service
     Loaded: loaded (/etc/systemd/system/promtail.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2024-04-22 22:14:38 JST; 2min 21s ago
   Main PID: 896077 (promtail)
      Tasks: 9 (limit: 9197)
     Memory: 24.5M
        CPU: 457ms
     CGroup: /system.slice/promtail.service
             └─896077 /usr/bin/promtail -config.file /etc/promtail/config.yml

ここまではまぁそりゃ何事も起きませんよねって感じ

Alice Rose 12日前

promtailのログがこれだけだから何も検知出来てない気がする

systemctl status promtail
● promtail.service - Promtail service
     Loaded: loaded (/etc/systemd/system/promtail.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2024-04-22 23:01:13 JST; 2s ago
   Main PID: 933243 (promtail)
      Tasks: 10 (limit: 9197)
     Memory: 14.6M
        CPU: 85ms
     CGroup: /system.slice/promtail.service
             └─933243 /usr/bin/promtail -config.file /etc/promtail/config.yml

Apr 22 23:01:13 **** systemd[1]: Started Promtail service.
Apr 22 23:01:13 **** promtail[933243]: level=info ts=2024-04-22T14:01:13.575809964Z caller=promtail.go:133 msg="Reloading configuration file" md5sum=8fa52b5bb95b0a622d28713580570045
Apr 22 23:01:13 **** promtail[933243]: level=info ts=2024-04-22T14:01:13.577414343Z caller=server.go:354 msg="server listening on addresses" http=[::]:9080 grpc=[::]:35145
Apr 22 23:01:13 **** promtail[933243]: level=info ts=2024-04-22T14:01:13.57765423Z caller=main.go:173 msg="Starting Promtail" version="(version=3.0.0, branch=release-3.0.x, revision=b4f71>
Apr 22 23:01:13 **** promtail[933243]: level=warn ts=2024-04-22T14:01:13.577814733Z caller=promtail.go:263 msg="enable watchConfig"

Alice Rose 12日前

ログファイルを直接拾うようにしてみる

/etc/promtail/config.yaml

scrape_configs:
- job_name: 'log'
  static_configs:
  - targets:
    - localhost
    labels:
      job: varlogs
      __path__: /var/log/**/*.log

pathを/var/log/だけだとfile is a directoryと一生怒られていた

sudo systemctl daemon-reload
sudo systemctl restart promtail
systemctl status promtail
● promtail.service - Promtail service
     Loaded: loaded (/etc/systemd/system/promtail.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2024-04-22 23:06:28 JST; 6s ago
   Main PID: 937557 (promtail)
      Tasks: 10 (limit: 9197)
     Memory: 43.4M
        CPU: 1.063s
     CGroup: /system.slice/promtail.service
             └─937557 /usr/bin/promtail -config.file /etc/promtail/config.yml
Apr 22 23:06:33 **** promtail[937557]: ts=2024-04-22T14:06:33.434091431Z caller=log.go:168 level=info msg="Seeked /var/log/kern.log - &{Offset:0 Whence:0}"
Apr 22 23:06:33 **** promtail[937557]: level=info ts=2024-04-22T14:06:33.434119769Z caller=tailer.go:147 component=tailer msg="tail routine: started" path=/var/log/kern.log
Apr 22 23:06:33 **** promtail[937557]: level=info ts=2024-04-22T14:06:33.434174586Z caller=tailer.go:147 component=tailer msg="tail routine: started" path=/var/log/cloud-init.log
Apr 22 23:06:33 **** promtail[937557]: level=info ts=2024-04-22T14:06:33.434190406Z caller=tailer.go:147 component=tailer msg="tail routine: started" path=/var/log/cloud-init-output.log
Apr 22 23:06:33 **** promtail[937557]: ts=2024-04-22T14:06:33.434260285Z caller=log.go:168 level=info msg="Seeked /var/log/cloud-init.log - &{Offset:0 Whence:0}"
Apr 22 23:06:33 **** promtail[937557]: level=info ts=2024-04-22T14:06:33.434366478Z caller=tailer.go:147 component=tailer msg="tail routine: started" path=/var/log/ubuntu-advantage.log
Apr 22 23:06:33 **** promtail[937557]: ts=2024-04-22T14:06:33.434419146Z caller=log.go:168 level=info msg="Seeked /var/log/ufw.log - &{Offset:0 Whence:0}"
Apr 22 23:06:33 **** promtail[937557]: ts=2024-04-22T14:06:33.434427834Z caller=log.go:168 level=info msg="Seeked /var/log/cron.log - &{Offset:0 Whence:0}"
Apr 22 23:06:33 **** promtail[937557]: level=info ts=2024-04-22T14:06:33.43462439Z caller=tailer.go:147 component=tailer msg="tail routine: started" path=/var/log/ufw.log
Apr 22 23:06:33 **** promtail[937557]: level=info ts=2024-04-22T14:06:33.434646586Z caller=tailer.go:147 component=tailer msg="tail routine: started" path=/var/log/cron.log

logファイルは拾えるようになったみたい

Alice Rose 12日前

あとはやっぱりジャーナルを拾ってほしい

Alice Rose 12日前

基本に返る

/etc/promtail/config.yaml

scrape_configs:
- job_name: 'log'
  static_configs:
  - targets:
    - localhost
    labels:
      job: varlogs
      __path__: /var/log/**/*.log
- job_name: systemd-journal
  journal:
    labels:
      cluster: ops-tools1
      job: default/systemd-journal
    path: /var/log/journal
  relabel_configs:
  - source_labels:
    - __journal__systemd_unit
    target_label: systemd_unit
  - source_labels:
    - __journal__hostname
    target_label: nodename
  - source_labels:
    - __journal_syslog_identifier
    target_label: syslog_identifier

Alice Rose 12日前

ふと見てみたら何か権限が家出したらしくエラー吐いてた

journalctl -u promtail -f
Apr 23 00:32:21 **** promtail[1000507]: level=error ts=2024-04-22T15:32:20.948096407Z caller=filetarget.go:385 msg="failed to start tailer" error="open /var/log/grafana/grafana.log: permission denied" filename=/var/log/grafana/grafana.log
Apr 23 00:32:31 **** promtail[1000507]: level=error ts=2024-04-22T15:32:30.948092707Z caller=filetarget.go:385 msg="failed to start tailer" error="open /var/log/grafana/grafana.log: permission denied" filename=/var/log/grafana/grafana.log
Apr 23 00:32:41 **** promtail[1000507]: level=error ts=2024-04-22T15:32:40.951417912Z caller=filetarget.go:385 msg="failed to start tailer" error="open /var/log/grafana/grafana.log: permission denied" filename=/var/log/grafana/grafana.log
Apr 23 00:32:51 **** promtail[1000507]: level=error ts=2024-04-22T15:32:50.9481548Z caller=filetarget.go:385 msg="failed to start tailer" error="open /var/log/grafana/grafana.log: permission denied" filename=/var/log/grafana/grafana.log
Apr 23 00:33:01 **** promtail[1000507]: level=error ts=2024-04-22T15:33:00.948094667Z caller=filetarget.go:385 msg="failed to start tailer" error="open /var/log/grafana/grafana.log: permission denied" filename=/var/log/grafana/grafana.log

Alice Rose 12日前

権限ないらしい
前に設定しなかったっけ？

getfacl /var/log/grafana/grafana.log
getfacl: Removing leading '/' from absolute path names
# file: var/log/grafana/grafana.log
# owner: grafana
# group: grafana
user::rw-
group::r--
other::---

Alice Rose 12日前

した

sudo setfacl -R -m u:promtail:rX /var/log

Seekedになったので読めるようになったみたい

journalctl -u promtail -f
Apr 23 00:33:51 **** promtail[1000507]: level=error ts=2024-04-22T15:33:50.947784698Z caller=filetarget.go:385 msg="failed to start tailer" error="open /var/log/grafana/grafana.log: permission denied" filename=/var/log/grafana/grafana.log
Apr 23 00:34:01 **** promtail[1000507]: level=error ts=2024-04-22T15:34:00.948279223Z caller=filetarget.go:385 msg="failed to start tailer" error="open /var/log/grafana/grafana.log: permission denied" filename=/var/log/grafana/grafana.log
Apr 23 00:34:11 **** promtail[1000507]: level=error ts=2024-04-22T15:34:10.947726713Z caller=filetarget.go:385 msg="failed to start tailer" error="open /var/log/grafana/grafana.log: permission denied" filename=/var/log/grafana/grafana.log
Apr 23 00:34:21 **** promtail[1000507]: level=error ts=2024-04-22T15:34:20.948002207Z caller=filetarget.go:385 msg="failed to start tailer" error="open /var/log/grafana/grafana.log: permission denied" filename=/var/log/grafana/grafana.log
Apr 23 00:34:31 **** promtail[1000507]: level=error ts=2024-04-22T15:34:30.948415174Z caller=filetarget.go:385 msg="failed to start tailer" error="open /var/log/grafana/grafana.log: permission denied" filename=/var/log/grafana/grafana.log
Apr 23 00:34:41 **** promtail[1000507]: level=error ts=2024-04-22T15:34:40.948059296Z caller=filetarget.go:385 msg="failed to start tailer" error="open /var/log/grafana/grafana.log: permission denied" filename=/var/log/grafana/grafana.log
Apr 23 00:34:51 **** promtail[1000507]: level=error ts=2024-04-22T15:34:50.951366228Z caller=filetarget.go:385 msg="failed to start tailer" error="open /var/log/grafana/grafana.log: permission denied" filename=/var/log/grafana/grafana.log
Apr 23 00:35:01 **** promtail[1000507]: level=error ts=2024-04-22T15:35:00.948646124Z caller=filetarget.go:385 msg="failed to start tailer" error="open /var/log/grafana/grafana.log: permission denied" filename=/var/log/grafana/grafana.log
Apr 23 00:35:11 **** promtail[1000507]: level=info ts=2024-04-22T15:35:10.948344198Z caller=tailer.go:147 component=tailer msg="tail routine: started" path=/var/log/grafana/grafana.log
Apr 23 00:35:11 **** promtail[1000507]: ts=2024-04-22T15:35:10.948533695Z caller=log.go:168 level=info msg="Seeked /var/log/grafana/grafana.log - &{Offset:0 Whence:0}"
Apr 23 00:36:50 **** promtail[1000507]: level=info ts=2024-04-22T15:36:50.354821424Z caller=filetargetmanager.go:192 msg="received file watcher event" name=/var/log/apt/eipp.log.xz op=CREATE
Apr 23 00:36:52 **** promtail[1000507]: level=info ts=2024-04-22T15:36:52.752294045Z caller=filetargetmanager.go:192 msg="received file watcher event" name=/var/log/apt/eipp.log.xz op=CREATE

Alice Rose 12日前

一応シンタックスも問題ないか確認して

promtail -config.file=/etc/promtail/config.yml -check-syntax
Valid config file! No syntax issues found

sudo systemctl daemon-reload
sudo systemctl restart promtail

一通りまた拾えるようになった

Alice Rose 12日前

治ったはいいけど結局何が原因で駄目になってて、何をしたから治った、があまり釈然としない
とても消化不良だが現段階だとどうにもなのでまた次再発したら考えることにする

終わり

このスクラップは12日前にクローズされました