diagram

仮説

凡例

• solid: usage
• dashed: DFD

モチベ

実験すれば良いけど、静的に解析したい。
→ code reading

repo

https://github.com/prometheus/prometheus/tree/main/storage/remote

https://github.com/prometheus/prometheus/tree/main/cmd/prometheus

ref

https://prometheus.io/docs/prometheus/latest/storage/#remote-storage-integrations

Remote storage integrations

https://prometheus.io/docs/prometheus/latest/images/remote_integrations.png

https://prometheus.io/docs/operating/integrations/#remote-endpoints-and-storage

Remote Endpoints and Storage

https://www.metricfire.com/blog/prometheus-remote-storage/

Remote read
When configured, Prometheus storage queries (e.g. via the HTTP API) are sent to both local and remote storage, and results are merged.
Note that to maintain reliability in the face of remote storage issues, alerting and recording rule evaluation use only the local TSDB.

code reading

あたりを つける

初期化:

https://github.com/prometheus/prometheus/blob/92d69803606620505ed8f0226abb640a121ec1cd/cmd/prometheus/main.go#L588-L593

https://github.com/prometheus/prometheus/blob/92d69803606620505ed8f0226abb640a121ec1cd/cmd/prometheus/main.go#L618

actor 登録:

https://github.com/prometheus/prometheus/blob/92d69803606620505ed8f0226abb640a121ec1cd/cmd/prometheus/main.go#L911-L933

始動:

https://github.com/prometheus/prometheus/blob/92d69803606620505ed8f0226abb640a121ec1cd/cmd/prometheus/main.go#L1158

lib

run.Group について

https://github.com/oklog/run

https://blog.gopheracademy.com/advent-2017/run-group/

定義

https://github.com/prometheus/prometheus/blob/92d69803606620505ed8f0226abb640a121ec1cd/storage/interface.go#L70-L81

https://github.com/prometheus/prometheus/blob/92d69803606620505ed8f0226abb640a121ec1cd/storage/interface.go#L56-L62

https://github.com/prometheus/prometheus/blob/92d69803606620505ed8f0226abb640a121ec1cd/storage/fanout.go#L37-L52

https://github.com/prometheus/prometheus/blob/92d69803606620505ed8f0226abb640a121ec1cd/scrape/manager.go#L103-L123

https://github.com/prometheus/prometheus/blob/92d69803606620505ed8f0226abb640a121ec1cd/scrape/manager.go#L145-L160

https://github.com/prometheus/prometheus/blob/92d69803606620505ed8f0226abb640a121ec1cd/scrape/scrape.go#L229-L254

https://github.com/prometheus/prometheus/blob/92d69803606620505ed8f0226abb640a121ec1cd/config/config.go#L457-L509

scraper main loop

scraper

reload

https://github.com/prometheus/prometheus/blob/92d69803606620505ed8f0226abb640a121ec1cd/scrape/manager.go#L162-L180

https://github.com/prometheus/prometheus/blob/92d69803606620505ed8f0226abb640a121ec1cd/scrape/manager.go#L182-L205

sync

https://github.com/prometheus/prometheus/blob/92d69803606620505ed8f0226abb640a121ec1cd/scrape/manager.go#L207

https://github.com/prometheus/prometheus/blob/92d69803606620505ed8f0226abb640a121ec1cd/scrape/manager.go#L226-L230

https://github.com/prometheus/prometheus/blob/92d69803606620505ed8f0226abb640a121ec1cd/scrape/scrape.go#L497-L534

https://github.com/prometheus/prometheus/blob/92d69803606620505ed8f0226abb640a121ec1cd/scrape/scrape.go#L536-L539

main loop

https://github.com/prometheus/prometheus/blob/92d69803606620505ed8f0226abb640a121ec1cd/scrape/scrape.go#L1239-L1300

write

https://github.com/prometheus/prometheus/blob/92d69803606620505ed8f0226abb640a121ec1cd/scrape/scrape.go#L1302-L1307

def

https://github.com/prometheus/prometheus/blob/92d69803606620505ed8f0226abb640a121ec1cd/scrape/scrape.go#L229-L254

https://github.com/prometheus/prometheus/blob/92d69803606620505ed8f0226abb640a121ec1cd/scrape/scrape.go#L872-L879

scrape interval

reload

https://github.com/prometheus/prometheus/blob/92d69803606620505ed8f0226abb640a121ec1cd/scrape/scrape.go#L422

sync

https://github.com/prometheus/prometheus/blob/92d69803606620505ed8f0226abb640a121ec1cd/scrape/scrape.go#L542

zero

https://github.com/prometheus/prometheus/blob/92d69803606620505ed8f0226abb640a121ec1cd/scrape/scrape.go#L1922-L1927

periodic exec

https://github.com/prometheus/prometheus/blob/92d69803606620505ed8f0226abb640a121ec1cd/scrape/scrape.go#L1251

https://pkg.go.dev/time#NewTicker

結局、書き込みインターバルが scrape interval だと言う証拠は、まだ分からない。

引き続き evaluate interval が remote read interval に相当するかを検証する。

global config

https://prometheus.io/docs/prometheus/latest/configuration/configuration/#configuration-file

global.evaluation_interval

rule config

https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/#rule_group

groups[].interval

https://github.com/SchemaStore/schemastore/blob/cfebb972126eedd47cc7b6ef8d2c2efc746df12e/src/schemas/json/prometheus.rules.json#L87-L101

eval loop

def

https://github.com/prometheus/prometheus/blob/92d69803606620505ed8f0226abb640a121ec1cd/config/config.go#L380-L393

https://github.com/prometheus/prometheus/blob/92d69803606620505ed8f0226abb640a121ec1cd/config/config.go#L387-L388

https://github.com/prometheus/prometheus/blob/92d69803606620505ed8f0226abb640a121ec1cd/config/config.go#L400-L401

main

https://github.com/prometheus/prometheus/blob/92d69803606620505ed8f0226abb640a121ec1cd/cmd/prometheus/main.go#L776-L802

https://github.com/prometheus/prometheus/blob/92d69803606620505ed8f0226abb640a121ec1cd/rules/manager.go#L1022-L1028

https://github.com/prometheus/prometheus/blob/92d69803606620505ed8f0226abb640a121ec1cd/cmd/prometheus/main.go#L1200-L1214

https://github.com/prometheus/prometheus/blob/92d69803606620505ed8f0226abb640a121ec1cd/cmd/prometheus/main.go#L650-L655

https://github.com/prometheus/prometheus/blob/92d69803606620505ed8f0226abb640a121ec1cd/rules/manager.go#L187-L190

engine

https://github.com/prometheus/prometheus/blob/92d69803606620505ed8f0226abb640a121ec1cd/promql/engine.go#L247-L260

https://github.com/prometheus/prometheus/blob/92d69803606620505ed8f0226abb640a121ec1cd/promql/engine.go#L279-L290

rules

https://github.com/prometheus/prometheus/blob/92d69803606620505ed8f0226abb640a121ec1cd/rules/manager.go#L953-L956

https://github.com/prometheus/prometheus/blob/92d69803606620505ed8f0226abb640a121ec1cd/rules/manager.go#L215-L222

https://github.com/prometheus/prometheus/blob/92d69803606620505ed8f0226abb640a121ec1cd/rules/manager.go#L184-L185

group

https://github.com/prometheus/prometheus/blob/92d69803606620505ed8f0226abb640a121ec1cd/rules/manager.go#L358

group interval

https://github.com/prometheus/prometheus/blob/92d69803606620505ed8f0226abb640a121ec1cd/rules/manager.go#L376-L380

loop

https://github.com/prometheus/prometheus/blob/92d69803606620505ed8f0226abb640a121ec1cd/rules/manager.go#L428-L447

rule group

https://github.com/prometheus/prometheus/blob/92d69803606620505ed8f0226abb640a121ec1cd/rules/manager.go#L1095-L1099

https://github.com/prometheus/prometheus/blob/92d69803606620505ed8f0226abb640a121ec1cd/model/rulefmt/rulefmt.go#L88-L95

ふりかえり

結局、いつ・どこで・だれが remote storage を fetch してんだよ。

readable storage

interface

https://github.com/prometheus/prometheus/blob/92d69803606620505ed8f0226abb640a121ec1cd/storage/interface.go#L70-L73

https://github.com/prometheus/prometheus/blob/92d69803606620505ed8f0226abb640a121ec1cd/storage/interface.go#L64-L68

https://github.com/prometheus/prometheus/blob/92d69803606620505ed8f0226abb640a121ec1cd/storage/interface.go#L90-L95

https://github.com/prometheus/prometheus/blob/92d69803606620505ed8f0226abb640a121ec1cd/storage/interface.go#L137-L142

read handler

only for API; ignore

https://pkg.go.dev/net/http#Handler

type Handler interface {
	ServeHTTP(ResponseWriter, *Request)
}

https://github.com/prometheus/prometheus/blob/92d69803606620505ed8f0226abb640a121ec1cd/storage/remote/read_handler.go#L44-L47

https://github.com/prometheus/prometheus/blob/92d69803606620505ed8f0226abb640a121ec1cd/storage/remote/read_handler.go#L33-L35

sampleAndChunkQueryableClient‎

https://github.com/prometheus/prometheus/blob/92d69803606620505ed8f0226abb640a121ec1cd/storage/remote/read.go#L33-L40

usage of queryable

https://github.com/prometheus/prometheus/blob/92d69803606620505ed8f0226abb640a121ec1cd/storage/remote/storage.go#L51-L62

https://github.com/prometheus/prometheus/blob/92d69803606620505ed8f0226abb640a121ec1cd/storage/merge.go#L70-L75

https://github.com/prometheus/prometheus/blob/92d69803606620505ed8f0226abb640a121ec1cd/storage/fanout.go#L37-L46