GrafanaでUbuntu Serverの監視体制をつくる

sudo apt purge grafana prometheus promtail loki
[sudo] password for ****:
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages were automatically installed and are no longer required:
  fonts-glyphicons-halflings libjs-bootstrap libjs-bootstrap4 libjs-d3 libjs-eonasdan-bootstrap-datetimepicker libjs-jquery-hotkeys libjs-moment libjs-moment-timezone libjs-mustache
  libjs-popper.js libjs-rickshaw libjs-sizzle musl node-jquery
Use 'sudo apt autoremove' to remove them.
The following packages will be REMOVED:
  grafana* loki* prometheus* promtail*
0 upgraded, 0 newly installed, 4 to remove and 17 not upgraded.
After this operation, 676 MB disk space will be freed.
Do you want to continue? [Y/n] y
(Reading database ... 159688 files and directories currently installed.)
Removing grafana (10.4.1) ...
Stopping and disabling grafana-server service...
Synchronizing state of grafana-server.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install disable grafana-server
Removed /etc/systemd/system/multi-user.target.wants/grafana-server.service.
Removing loki (3.0.0) ...
Removing prometheus (2.31.2+ds1-1ubuntu1.22.04.2) ...
Removing promtail (3.0.0) ...
Processing triggers for man-db (2.10.2-1) ...
(Reading database ... 147631 files and directories currently installed.)
Purging configuration files for prometheus (2.31.2+ds1-1ubuntu1.22.04.2) ...
Purging configuration files for loki (3.0.0) ...
dpkg: warning: while removing loki, directory '/etc/loki' not empty so not removed
Purging configuration files for promtail (3.0.0) ...
dpkg: warning: while removing promtail, directory '/etc/promtail' not empty so not removed
Purging configuration files for grafana (10.4.1) ...
dpkg: warning: while removing grafana, directory '/usr/lib/systemd/system' not empty so not removed
dpkg: warning: while removing grafana, directory '/etc/grafana' not empty so not removed

sudo apt autoremove
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
sh: 0: getcwd() failed: No such file or directory
The following packages will be REMOVED:
  fonts-glyphicons-halflings libjs-bootstrap libjs-bootstrap4 libjs-d3 libjs-eonasdan-bootstrap-datetimepicker libjs-jquery-hotkeys libjs-moment libjs-moment-timezone libjs-mustache
  libjs-popper.js libjs-rickshaw libjs-sizzle musl node-jquery
sh: 0: getcwd() failed: No such file or directory
0 upgraded, 0 newly installed, 14 to remove and 17 not upgraded.
sh: 0: getcwd() failed: No such file or directory
After this operation, 183 MB disk space will be freed.
Do you want to continue? [Y/n] y
sh: 0: getcwd() failed: No such file or directory
(Reading database ... 147612 files and directories currently installed.)
Removing libjs-eonasdan-bootstrap-datetimepicker (4.17.47-5) ...
Removing libjs-bootstrap (3.4.1+dfsg-2) ...
Removing fonts-glyphicons-halflings (1.009~3.4.1+dfsg-2) ...
dpkg: warning: while removing fonts-glyphicons-halflings, directory '/usr/share/fonts/truetype/glyphicons' not empty so not removed
Removing libjs-bootstrap4 (4.6.0+dfsg1-4) ...
Removing libjs-rickshaw (1.5.1.dfsg-5) ...
Removing libjs-d3 (3.5.17-4) ...
Removing libjs-jquery-hotkeys (0~20130707+git2d51e3a9+dfsg-2ubuntu1) ...
Removing libjs-moment-timezone (0.5.34+dfsg-1+2021e) ...
Removing libjs-moment (2.29.1+ds-3ubuntu0.2) ...
Removing libjs-mustache (2.3.2-2) ...
Removing libjs-popper.js (1.16.1+ds-5) ...
Removing libjs-sizzle (2.3.6+ds+~2.3.3-1) ...
Removing musl:amd64 (1.2.2-4) ...
Removing node-jquery (3.6.0+dfsg+~3.5.13-1) ...
Processing triggers for man-db (2.10.2-1) ...
Processing triggers for fontconfig (2.13.1-4.2ubuntu5) ...
sh: 0: getcwd() failed: No such file or directory

sudo apt purge prometheus-node-exporter

sudo apt install prometheus prometheus-node-exporter
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  fonts-glyphicons-halflings libio-pty-perl libipc-run-perl libjs-bootstrap libjs-bootstrap4 libjs-d3 libjs-eonasdan-bootstrap-datetimepicker libjs-jquery-hotkeys libjs-moment
  libjs-moment-timezone libjs-mustache libjs-popper.js libjs-rickshaw libjs-sizzle libtime-duration-perl moreutils node-jquery prometheus-node-exporter-collectors smartmontools
Suggested packages:
  gsmartcontrol smart-notifier
The following NEW packages will be installed:
  fonts-glyphicons-halflings libio-pty-perl libipc-run-perl libjs-bootstrap libjs-bootstrap4 libjs-d3 libjs-eonasdan-bootstrap-datetimepicker libjs-jquery-hotkeys libjs-moment
  libjs-moment-timezone libjs-mustache libjs-popper.js libjs-rickshaw libjs-sizzle libtime-duration-perl moreutils node-jquery prometheus prometheus-node-exporter
  prometheus-node-exporter-collectors smartmontools
0 upgraded, 21 newly installed, 0 to remove and 20 not upgraded.
Need to get 25.7 MB of archives.
After this operation, 280 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://jp.archive.ubuntu.com/ubuntu jammy/universe amd64 libjs-jquery-hotkeys all 0~20130707+git2d51e3a9+dfsg-2ubuntu1 [11.4 kB]
Get:2 http://jp.archive.ubuntu.com/ubuntu jammy/universe amd64 fonts-glyphicons-halflings all 1.009~3.4.1+dfsg-2 [117 kB]
Get:3 http://jp.archive.ubuntu.com/ubuntu jammy/universe amd64 libjs-popper.js all 1.16.1+ds-5 [53.8 kB]
Get:4 http://jp.archive.ubuntu.com/ubuntu jammy/universe amd64 libjs-bootstrap4 all 4.6.0+dfsg1-4 [534 kB]
Get:5 http://jp.archive.ubuntu.com/ubuntu jammy/universe amd64 libjs-bootstrap all 3.4.1+dfsg-2 [124 kB]
Get:6 http://jp.archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libjs-moment all 2.29.1+ds-3ubuntu0.2 [146 kB]
Get:7 http://jp.archive.ubuntu.com/ubuntu jammy/universe amd64 libjs-eonasdan-bootstrap-datetimepicker all 4.17.47-5 [28.4 kB]
Get:8 http://jp.archive.ubuntu.com/ubuntu jammy/universe amd64 libjs-moment-timezone all 0.5.34+dfsg-1+2021e [759 kB]
Get:9 http://jp.archive.ubuntu.com/ubuntu jammy/universe amd64 libjs-mustache all 2.3.2-2 [20.6 kB]
Get:10 http://jp.archive.ubuntu.com/ubuntu jammy/universe amd64 libjs-d3 all 3.5.17-4 [132 kB]
Get:11 http://jp.archive.ubuntu.com/ubuntu jammy/universe amd64 libjs-rickshaw all 1.5.1.dfsg-5 [646 kB]
Get:12 http://jp.archive.ubuntu.com/ubuntu jammy-updates/universe amd64 prometheus amd64 2.31.2+ds1-1ubuntu1.22.04.2 [18.3 MB]
Get:13 http://jp.archive.ubuntu.com/ubuntu jammy-updates/universe amd64 prometheus-node-exporter amd64 1.3.1-1ubuntu0.22.04.2 [3887 kB]
Get:14 http://jp.archive.ubuntu.com/ubuntu jammy/main amd64 libio-pty-perl amd64 1:1.15-2build2 [35.1 kB]
Get:15 http://jp.archive.ubuntu.com/ubuntu jammy/main amd64 libipc-run-perl all 20200505.0-1 [89.8 kB]
Get:16 http://jp.archive.ubuntu.com/ubuntu jammy/universe amd64 libjs-sizzle all 2.3.6+ds+~2.3.3-1 [32.3 kB]
Get:17 http://jp.archive.ubuntu.com/ubuntu jammy/main amd64 libtime-duration-perl all 1.21-1 [13.1 kB]
Get:18 http://jp.archive.ubuntu.com/ubuntu jammy/universe amd64 moreutils amd64 0.66-1 [63.3 kB]
Get:19 http://jp.archive.ubuntu.com/ubuntu jammy/universe amd64 node-jquery all 3.6.0+dfsg+~3.5.13-1 [160 kB]
Get:20 http://jp.archive.ubuntu.com/ubuntu jammy/universe amd64 prometheus-node-exporter-collectors all 0+git20211024.8eeeffb-1 [27.1 kB]
Get:21 http://jp.archive.ubuntu.com/ubuntu jammy-updates/main amd64 smartmontools amd64 7.2-1ubuntu0.1 [583 kB]
Fetched 25.7 MB in 8s (3136 kB/s)
Selecting previously unselected package libjs-jquery-hotkeys.
(Reading database ... 146488 files and directories currently installed.)
Preparing to unpack .../00-libjs-jquery-hotkeys_0~20130707+git2d51e3a9+dfsg-2ubuntu1_all.deb ...
Unpacking libjs-jquery-hotkeys (0~20130707+git2d51e3a9+dfsg-2ubuntu1) ...
Selecting previously unselected package fonts-glyphicons-halflings.
Preparing to unpack .../01-fonts-glyphicons-halflings_1.009~3.4.1+dfsg-2_all.deb ...
Unpacking fonts-glyphicons-halflings (1.009~3.4.1+dfsg-2) ...
Selecting previously unselected package libjs-popper.js.
Preparing to unpack .../02-libjs-popper.js_1.16.1+ds-5_all.deb ...
Unpacking libjs-popper.js (1.16.1+ds-5) ...
Selecting previously unselected package libjs-bootstrap4.
Preparing to unpack .../03-libjs-bootstrap4_4.6.0+dfsg1-4_all.deb ...
Unpacking libjs-bootstrap4 (4.6.0+dfsg1-4) ...
Selecting previously unselected package libjs-bootstrap.
Preparing to unpack .../04-libjs-bootstrap_3.4.1+dfsg-2_all.deb ...
Unpacking libjs-bootstrap (3.4.1+dfsg-2) ...
Selecting previously unselected package libjs-moment.
Preparing to unpack .../05-libjs-moment_2.29.1+ds-3ubuntu0.2_all.deb ...
Unpacking libjs-moment (2.29.1+ds-3ubuntu0.2) ...
Preparing to unpack .../06-libjs-eonasdan-bootstrap-datetimepicker_4.17.47-5_all.deb ...
Progress: [ 14%] [#######################............................................................................................................................................]
Unpacking libjs-eonasdan-bootstrap-datetimepicker (4.17.47-5) ...
Selecting previously unselected package libjs-moment-timezone........................................................................................................................]
Preparing to unpack .../07-libjs-moment-timezone_0.5.34+dfsg-1+2021e_all.deb ........................................................................................................]
Unpacking libjs-moment-timezone (0.5.34+dfsg-1+2021e) ...
Selecting previously unselected package libjs-mustache...............................................................................................................................]
Preparing to unpack .../08-libjs-mustache_2.3.2-2_all.deb ...........................................................................................................................]
Unpacking libjs-mustache (2.3.2-2) ...
Selecting previously unselected package libjs-d3.#...................................................................................................................................]
Preparing to unpack .../09-libjs-d3_3.5.17-4_all.deb ................................................................................................................................]
Unpacking libjs-d3 (3.5.17-4) ...
Selecting previously unselected package libjs-rickshaw...............................................................................................................................]
Preparing to unpack .../10-libjs-rickshaw_1.5.1.dfsg-5_all.deb ......................................................................................................................]
Unpacking libjs-rickshaw (1.5.1.dfsg-5) ...
Selecting previously unselected package prometheus.#######...........................................................................................................................]
Preparing to unpack .../11-prometheus_2.31.2+ds1-1ubuntu1.22.04.2_amd64.deb .........................................................................................................]
Unpacking prometheus (2.31.2+ds1-1ubuntu1.22.04.2) ...
Selecting previously unselected package prometheus-node-exporter.....................................................................................................................]
Preparing to unpack .../12-prometheus-node-exporter_1.3.1-1ubuntu0.22.04.2_amd64.deb ................................................................................................]
Unpacking prometheus-node-exporter (1.3.1-1ubuntu0.22.04.2) ...
Selecting previously unselected package libio-pty-perl.##########....................................................................................................................]
Preparing to unpack .../13-libio-pty-perl_1%3a1.15-2build2_amd64.deb ................................................................................................................]
Unpacking libio-pty-perl (1:1.15-2build2) ...
Selecting previously unselected package libipc-run-perl.#############................................................................................................................]
Preparing to unpack .../14-libipc-run-perl_20200505.0-1_all.deb ...####..............................................................................................................]
Unpacking libipc-run-perl (20200505.0-1) ...
Selecting previously unselected package libjs-sizzle.####################............................................................................................................]
Preparing to unpack .../15-libjs-sizzle_2.3.6+ds+~2.3.3-1_all.deb ...######..........................................................................................................]
Unpacking libjs-sizzle (2.3.6+ds+~2.3.3-1) ...
Selecting previously unselected package libtime-duration-perl.###############........................................................................................................]
Preparing to unpack .../16-libtime-duration-perl_1.21-1_all.deb ...############......................................................................................................]
Unpacking libtime-duration-perl (1.21-1) ...
Selecting previously unselected package moreutils.###############################....................................................................................................]
Preparing to unpack .../17-moreutils_0.66-1_amd64.deb ...##########################..................................................................................................]
Unpacking moreutils (0.66-1) ...
Selecting previously unselected package node-jquery.#################################................................................................................................]
Preparing to unpack .../18-node-jquery_3.6.0+dfsg+~3.5.13-1_all.deb ...################..............................................................................................]
Unpacking node-jquery (3.6.0+dfsg+~3.5.13-1) ...
Selecting previously unselected package prometheus-node-exporter-collectors.############.............................................................................................]
Preparing to unpack .../19-prometheus-node-exporter-collectors_0+git20211024.8eeeffb-1_all.deb ......................................................................................]
Unpacking prometheus-node-exporter-collectors (0+git20211024.8eeeffb-1) ...
Selecting previously unselected package smartmontools.######################################.........................................................................................]
Preparing to unpack .../20-smartmontools_7.2-1ubuntu0.1_amd64.deb ...#########################.......................................................................................]
Unpacking smartmontools (7.2-1ubuntu0.1) ...
Setting up smartmontools (7.2-1ubuntu0.1) ...###################################################.....................................................................................]
Setting up libjs-popper.js (1.16.1+ds-5) ...######################################################...................................................................................]
Setting up libio-pty-perl (1:1.15-2build2) ...########################################################...............................................................................]
Setting up fonts-glyphicons-halflings (1.009~3.4.1+dfsg-2) ...############################################...........................................................................]
Setting up libjs-moment (2.29.1+ds-3ubuntu0.2) ...############################################################.......................................................................]
Setting up libjs-sizzle (2.3.6+ds+~2.3.3-1) ...##################################################################....................................................................]
Setting up prometheus-node-exporter (1.3.1-1ubuntu0.22.04.2) ...#####################################################................................................................]
Created symlink /etc/systemd/system/multi-user.target.wants/prometheus-node-exporter.service → /lib/systemd/system/prometheus-node-exporter.service..................................]
Setting up libjs-d3 (3.5.17-4) ...#########################################################################################..........................................................]
Setting up libjs-moment-timezone (0.5.34+dfsg-1+2021e) ...###################################################################........................................................]
Setting up libipc-run-perl (20200505.0-1) ...####################################################################################....................................................]
Setting up libtime-duration-perl (1.21-1) ...########################################################################################................................................]
Setting up libjs-bootstrap4 (4.6.0+dfsg1-4) ...#########################################################################################.............................................]
Setting up libjs-mustache (2.3.2-2) ...#####################################################################################################.........................................]
Setting up libjs-jquery-hotkeys (0~20130707+git2d51e3a9+dfsg-2ubuntu1) ...######################################################################.....................................]
Setting up node-jquery (3.6.0+dfsg+~3.5.13-1) ...###################################################################################################.................................]
Setting up libjs-bootstrap (3.4.1+dfsg-2) ...###########################################################################################################.............................]
Setting up libjs-eonasdan-bootstrap-datetimepicker (4.17.47-5) ...##########################################################################################.........................]
Setting up moreutils (0.66-1) ...##############################################################################################################################......................]
Setting up libjs-rickshaw (1.5.1.dfsg-5) ...#######################################################################################################################..................]
Setting up prometheus-node-exporter-collectors (0+git20211024.8eeeffb-1) ...###########################################################################################..............]
Created symlink /etc/systemd/system/timers.target.wants/prometheus-node-exporter-apt.timer → /lib/systemd/system/prometheus-node-exporter-apt.timer.#######################..........]
Created symlink /etc/systemd/system/timers.target.wants/prometheus-node-exporter-ipmitool-sensor.timer → /lib/systemd/system/prometheus-node-exporter-ipmitool-sensor.timer.#........]
Created symlink /etc/systemd/system/timers.target.wants/prometheus-node-exporter-mellanox-hca-temp.timer → /lib/systemd/system/prometheus-node-exporter-mellanox-hca-temp.timer.
Created symlink /etc/systemd/system/timers.target.wants/prometheus-node-exporter-nvme.timer → /lib/systemd/system/prometheus-node-exporter-nvme.timer.
Created symlink /etc/systemd/system/timers.target.wants/prometheus-node-exporter-smartmon.timer → /lib/systemd/system/prometheus-node-exporter-smartmon.timer.
Setting up prometheus (2.31.2+ds1-1ubuntu1.22.04.2) ...
Created symlink /etc/systemd/system/multi-user.target.wants/prometheus.service → /lib/systemd/system/prometheus.service.#######################################################......]
Processing triggers for fontconfig (2.13.1-4.2ubuntu5) ...#######################################################################################################################....]
Processing triggers for man-db (2.10.2-1) ...######################################################################################################################################..]
Scanning processes...
Scanning processor microcode...
Scanning linux images...

Running kernel seems to be up-to-date.

The processor microcode seems to be up-to-date.

No services need to be restarted.

No containers need to be restarted.

No user sessions are running outdated binaries.

No VM guests are running outdated hypervisor (qemu) binaries on this host.

sudo systemctl enable prometheus prometheus-node-exporter
Synchronizing state of prometheus.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable prometheus
Synchronizing state of prometheus-node-exporter.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable prometheus-node-exporter

sudo ufw allow 9090
Rule added
Rule added (v6)

sudo apt install grafana
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  musl
The following NEW packages will be installed:
  grafana musl
0 upgraded, 2 newly installed, 0 to remove and 20 not upgraded.
Need to get 115 MB of archives.
After this operation, 424 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 https://apt.grafana.com stable/main amd64 grafana amd64 10.4.2 [114 MB]
Get:2 http://jp.archive.ubuntu.com/ubuntu jammy/universe amd64 musl amd64 1.2.2-4 [407 kB]
Fetched 115 MB in 2s (54.1 MB/s)
Selecting previously unselected package musl:amd64.
(Reading database ... 147717 files and directories currently installed.)
Preparing to unpack .../musl_1.2.2-4_amd64.deb ...
Unpacking musl:amd64 (1.2.2-4) ...
Selecting previously unselected package grafana.
Preparing to unpack .../grafana_10.4.2_amd64.deb ...
Unpacking grafana (10.4.2) ...
Setting up musl:amd64 (1.2.2-4) ...
Setting up grafana (10.4.2) ...
### NOT starting on installation, please execute the following statements to configure grafana to start automatically using systemd
 sudo /bin/systemctl daemon-reload
 sudo /bin/systemctl enable grafana-server
### You can start grafana-server by executing
 sudo /bin/systemctl start grafana-server
Processing triggers for man-db (2.10.2-1) ...
Scanning processes...
Scanning processor microcode...
Scanning linux images...

...
#################################### Server ####################################
[server]
# Protocol (http, https, h2, socket)
;protocol = http

# This is the minimum TLS version allowed. By default, this value is empty. Accepted values are: TLS1.2, TLS1.3. If nothing is set TLS1.2 would be taken
;min_tls_version = ""

# The ip address to bind to, empty will bind to all interfaces
;http_addr =

# The http port  to use
- ;http_port = 3000
+ http_port = 3333

# The public facing domain name used to access grafana from a browser
;domain = localhost
...

+ grafana.***.*** {
+         reverse_proxy localhost:3333
+ }

sudo systemctl restart caddy

sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable grafana-server
Synchronizing state of grafana-server.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable grafana-server
Created symlink /etc/systemd/system/multi-user.target.wants/grafana-server.service → /lib/systemd/system/grafana-server.service.
sudo /bin/systemctl start grafana-server
sudo /bin/systemctl status grafana-server
● grafana-server.service - Grafana instance
     Loaded: loaded (/lib/systemd/system/grafana-server.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2024-04-18 02:03:35 JST; 12s ago
       Docs: http://docs.grafana.org
   Main PID: 35491 (grafana)
      Tasks: 13 (limit: 9198)
     Memory: 45.2M
        CPU: 1.768s
     CGroup: /system.slice/grafana-server.service
             └─35491 /usr/share/grafana/bin/grafana server --config=/etc/grafana/grafana.ini --pidfile=/run/grafana/grafana-server.pid --packaging=deb cfg:default.paths.logs=/var/log/g>

sudo apt install loki promtail
[sudo] password for ****:
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following NEW packages will be installed:
  loki promtail
0 upgraded, 2 newly installed, 0 to remove and 18 not upgraded.
Need to get 51.2 MB of archives.
After this operation, 169 MB of additional disk space will be used.
Get:1 https://apt.grafana.com stable/main amd64 loki amd64 3.0.0 [22.6 MB]
Get:2 https://apt.grafana.com stable/main amd64 promtail amd64 3.0.0 [28.6 MB]
Fetched 51.2 MB in 5s (11.4 MB/s)
Selecting previously unselected package loki.
(Reading database ... 159690 files and directories currently installed.)
Preparing to unpack .../archives/loki_3.0.0_amd64.deb ...
Unpacking loki (3.0.0) ...
Selecting previously unselected package promtail.
Preparing to unpack .../promtail_3.0.0_amd64.deb ...
Unpacking promtail (3.0.0) ...
Setting up loki (3.0.0) ...
 Post Install of a clean install
 Reload the service unit from disk
 Unmask the service
 Set the preset flag for the service unit
 Set the enabled flag for the service unit
Setting up promtail (3.0.0) ...
 Post Install of a clean install
 Reload the service unit from disk
 Unmask the service
 Set the preset flag for the service unit
 Set the enabled flag for the service unit
Scanning processes...
Scanning processor microcode...
Scanning linux images...

systemctl status promtail
● promtail.service - Promtail service
     Loaded: loaded (/etc/systemd/system/promtail.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2024-04-18 12:13:37 JST; 45s ago
   Main PID: 489421 (promtail)
      Tasks: 8 (limit: 9198)
     Memory: 14.9M
        CPU: 349ms
     CGroup: /system.slice/promtail.service
             └─489421 /usr/bin/promtail -config.file /etc/promtail/config.yml

systemctl status loki
● loki.service - Loki service
     Loaded: loaded (/etc/systemd/system/loki.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2024-04-18 12:13:35 JST; 1min 16s ago
   Main PID: 489197 (loki)
      Tasks: 10 (limit: 9198)
     Memory: 29.3M
        CPU: 888ms
     CGroup: /system.slice/loki.service
             └─489197 /usr/bin/loki -config.file /etc/loki/config.yml

scrape_configs:
  - job_name: journal
    journal:
      json: false
      max_age: 12h
      path: /var/log/journal
      matches: _TRANSPORT=kernel
      labels:
        job: systemd-journal
    relabel_configs:
      - source_labels: ['__journal__systemd_unit']
        target_label: 'unit'

- job_name: systemd-journal
  journal:
    labels:
      cluster: ops-tools1
      job: default/systemd-journal
    path: /var/log/journal
  relabel_configs:
  - source_labels:
    - __journal__systemd_unit
    target_label: systemd_unit
  - source_labels:
    - __journal__hostname
    target_label: nodename
  - source_labels:
    - __journal_syslog_identifier
    target_label: syslog_identifier