Closed11
ossf/scorecard 使ってみる
Futa Hirakobaインストール方法が見つからない。go でできてるので go install で入れてみる。
❯ go install github.com/ossf/scorecard@latest
入った。
❯ scorecard version
Error: unknown command "version" for "./scorecard"
Run './scorecard --help' for usage.
unknown command "version" for "./scorecard"
バージョン確認しようとしたがまさかのバージョンはコマンドに含まれてなかった。
Futa Hirakoba試しに自信を評価させてみる。
❯ scorecard --repo=https://github.com/ossf/scorecard
Starting [Active]
Starting [Branch-Protection]
Starting [CI-Tests]
Starting [CII-Best-Practices]
Starting [Code-Review]
Starting [Contributors]
Starting [Frozen-Deps]
Starting [Fuzzing]
Starting [Packaging]
Starting [Pull-Requests]
Starting [SAST]
Starting [Security-Policy]
Starting [Signed-Releases]
Starting [Signed-Tags]
{"level":"warn","ts":1625650068.096907,"caller":"roundtripper/roundtripper.go:173","msg":"Rate limit exceeded. Waiting 15m20.903105s to retry..."}
{"level":"warn","ts":1625650068.097593,"caller":"roundtripper/roundtripper.go:173","msg":"Rate limit exceeded. Waiting 15m20.90241s to retry..."}
{"level":"warn","ts":1625650068.097617,"caller":"roundtripper/roundtripper.go:173","msg":"Rate limit exceeded. Waiting 15m20.902415s to retry..."}
{"level":"warn","ts":1625650068.251936,"caller":"roundtripper/roundtripper.go:173","msg":"Rate limit exceeded. Waiting 59m59.7481s to retry..."}
{"level":"warn","ts":1625650068.291726,"caller":"roundtripper/roundtripper.go:173","msg":"Rate limit exceeded. Waiting 15m20.708281s to retry..."}
{"level":"warn","ts":1625650068.3224268,"caller":"roundtripper/roundtripper.go:173","msg":"Rate limit exceeded. Waiting 15m20.677579s to retry..."}
{"level":"warn","ts":1625650068.322602,"caller":"roundtripper/roundtripper.go:173","msg":"Rate limit exceeded. Waiting 15m20.677401s to retry..."}
Finished [Fuzzing]
Finished [SAST]
{"level":"warn","ts":1625650068.3705778,"caller":"roundtripper/roundtripper.go:173","msg":"Rate limit exceeded. Waiting 15m20.629464s to retry..."}
{"level":"warn","ts":1625650068.388341,"caller":"roundtripper/roundtripper.go:173","msg":"Rate limit exceeded. Waiting 15m20.611666s to retry..."}
Finished [Security-Policy]
Finished [Frozen-Deps]
Finished [CII-Best-Practices]
^C
めちゃRate limit exceededで怒られた。
そしてFinished [CII-Best-Practices]から動かなくなった。
Futa Hirakobaどうやら GitHub の API を叩きまくるから Personal Access Token で認証しないとすぐ rate limit に到達してしまうらしい。
Before running Scorecard, you need to, either:
create a GitHub access token and set it in environment variable GITHUB_AUTH_TOKEN. This helps to avoid the GitHub's api rate limits with unauthenticated requests.
Authentication
なんの権限も付与していないPATを作成した
Futa Hirakobaおーちゃんと動いた。
GITHUB_AUTH_TOKEN=xxxx scorecard --repo=https://github.com/ossf/scorecard
Starting [Active]
Starting [Branch-Protection]
Starting [CI-Tests]
Starting [CII-Best-Practices]
Starting [Code-Review]
Starting [Contributors]
Starting [Frozen-Deps]
Starting [Fuzzing]
Starting [Packaging]
Starting [Pull-Requests]
Starting [SAST]
Starting [Security-Policy]
Starting [Signed-Releases]
Starting [Signed-Tags]
{"level":"info","ts":1625650300.342906,"caller":"roundtripper/roundtripper.go:65","msg":"using token 0 of total 1."}
{"level":"info","ts":1625650300.342999,"caller":"roundtripper/roundtripper.go:65","msg":"using token 0 of total 1."}
{"level":"info","ts":1625650300.3431182,"caller":"roundtripper/roundtripper.go:65","msg":"using token 0 of total 1."}
{"level":"info","ts":1625650300.343125,"caller":"roundtripper/roundtripper.go:65","msg":"using token 0 of total 1."}
{"level":"info","ts":1625650300.343206,"caller":"roundtripper/roundtripper.go:65","msg":"using token 0 of total 1."}
{"level":"info","ts":1625650300.343265,"caller":"roundtripper/roundtripper.go:65","msg":"using token 0 of total 1."}
{"level":"info","ts":1625650300.3433862,"caller":"roundtripper/roundtripper.go:65","msg":"using token 0 of total 1."}
{"level":"info","ts":1625650300.343329,"caller":"roundtripper/roundtripper.go:65","msg":"using token 0 of total 1."}
{"level":"info","ts":1625650300.343426,"caller":"roundtripper/roundtripper.go:65","msg":"using token 0 of total 1."}
{"level":"info","ts":1625650300.342905,"caller":"roundtripper/roundtripper.go:65","msg":"using token 0 of total 1."}
{"level":"info","ts":1625650300.3431401,"caller":"roundtripper/roundtripper.go:65","msg":"using token 0 of total 1."}
{"level":"info","ts":1625650300.3444328,"caller":"roundtripper/roundtripper.go:65","msg":"using token 0 of total 1."}
{"level":"info","ts":1625650300.344446,"caller":"roundtripper/roundtripper.go:65","msg":"using token 0 of total 1."}
{"level":"info","ts":1625650300.344953,"caller":"roundtripper/roundtripper.go:65","msg":"using token 0 of total 1."}
Finished [Fuzzing]
Finished [SAST]
Finished [Security-Policy]
Finished [CII-Best-Practices]
Finished [Frozen-Deps]
Finished [Branch-Protection]
Finished [Signed-Tags]
Finished [Signed-Releases]
Finished [Packaging]
Finished [Contributors]
Finished [Active]
Finished [Pull-Requests]
Finished [Code-Review]
Finished [CI-Tests]
RESULTS
-------
Active: Pass 10
Branch-Protection: Fail 0
CI-Tests: Pass 10
CII-Best-Practices: Fail 10
Code-Review: Pass 10
Contributors: Pass 10
Frozen-Deps: Pass 10
Fuzzing: Fail 10
Packaging: Pass 10
Pull-Requests: Pass 10
SAST: Pass 10
Security-Policy: Pass 10
Signed-Releases: Fail 3
Signed-Tags: Fail 4
このままじゃよくわからない。--show-detailsしてみる。
前半は省略
RESULTS
-------
Active: Pass 10
commits in last 90 days: 30
Branch-Protection: Fail 0
error, retrying: GET https://api.github.com/repos/ossf/scorecard/branches/main/protection: 404 Not Found []
CI-Tests: Pass 10
CI test found: pr: 663, context: github-actionsuccess, url: https://api.github.com/repos/ossf/scorecard/check-runs/2998449248uccess
CI test found: pr: 658, context: github-actionsuccess, url: https://api.github.com/repos/ossf/scorecard/check-runs/2988216073uccess
CI test found: pr: 656, context: github-actionsuccess, url: https://api.github.com/repos/ossf/scorecard/check-runs/2984995844uccess
CI test found: pr: 654, context: github-actionsuccess, url: https://api.github.com/repos/ossf/scorecard/check-runs/2985243874uccess
CI test found: pr: 652, context: github-actionsuccess, url: https://api.github.com/repos/ossf/scorecard/check-runs/2985208338uccess
CI test found: pr: 649, context: github-actionsuccess, url: https://api.github.com/repos/ossf/scorecard/check-runs/2975804273uccess
CI test found: pr: 643, context: github-actionsuccess, url: https://api.github.com/repos/ossf/scorecard/check-runs/2970564487uccess
CI test found: pr: 641, context: github-actionsuccess, url: https://api.github.com/repos/ossf/scorecard/check-runs/2967571364uccess
CI test found: pr: 640, context: github-actionsuccess, url: https://api.github.com/repos/ossf/scorecard/check-runs/2967020555uccess
CI test found: pr: 637, context: github-actionsuccess, url: https://api.github.com/repos/ossf/scorecard/check-runs/2959402000uccess
CI test found: pr: 635, context: github-actionsuccess, url: https://api.github.com/repos/ossf/scorecard/check-runs/2955448364uccess
CI test found: pr: 634, context: github-actionsuccess, url: https://api.github.com/repos/ossf/scorecard/check-runs/2945495569uccess
CI test found: pr: 633, context: github-actionsuccess, url: https://api.github.com/repos/ossf/scorecard/check-runs/2945018883uccess
CI test found: pr: 628, context: github-actionsuccess, url: https://api.github.com/repos/ossf/scorecard/check-runs/2938130596uccess
CI test found: pr: 627, context: github-actionsuccess, url: https://api.github.com/repos/ossf/scorecard/check-runs/2944692090uccess
CI test found: pr: 626, context: github-actionsuccess, url: https://api.github.com/repos/ossf/scorecard/check-runs/2938835879uccess
CI test found: pr: 625, context: github-actionsuccess, url: https://api.github.com/repos/ossf/scorecard/check-runs/2936738100uccess
CI test found: pr: 624, context: github-actionsuccess, url: https://api.github.com/repos/ossf/scorecard/check-runs/2943644563uccess
CI test found: pr: 623, context: github-actionsuccess, url: https://api.github.com/repos/ossf/scorecard/check-runs/2944501189uccess
CI test found: pr: 622, context: github-actionsuccess, url: https://api.github.com/repos/ossf/scorecard/check-runs/2935636381uccess
CI test found: pr: 621, context: github-actionsuccess, url: https://api.github.com/repos/ossf/scorecard/check-runs/2935956732uccess
CI test found: pr: 620, context: github-actionsuccess, url: https://api.github.com/repos/ossf/scorecard/check-runs/2928721054uccess
CI test found: pr: 616, context: github-actionsuccess, url: https://api.github.com/repos/ossf/scorecard/check-runs/2924458994uccess
CI test found: pr: 614, context: github-actionsuccess, url: https://api.github.com/repos/ossf/scorecard/check-runs/2936436628uccess
CI test found: pr: 612, context: github-actionsuccess, url: https://api.github.com/repos/ossf/scorecard/check-runs/2909709728uccess
CI test found: pr: 610, context: github-actionsuccess, url: https://api.github.com/repos/ossf/scorecard/check-runs/2916273401uccess
CI test found: pr: 606, context: github-actionsuccess, url: https://api.github.com/repos/ossf/scorecard/check-runs/2907147171uccess
CI test found: pr: 605, context: github-actionsuccess, url: https://api.github.com/repos/ossf/scorecard/check-runs/2899455987uccess
found CI tests for 28 of 28 merged PRs
CII-Best-Practices: Fail 10
no badge found
Code-Review: Pass 10
found review approved pr: 663
found review approved pr: 658
found review approved pr: 656
found review approved pr: 654
found review approved pr: 652
found review approved pr: 649
found review approved pr: 643
found review approved pr: 641
found review approved pr: 640
found review approved pr: 637
found review approved pr: 635
found review approved pr: 634
found review approved pr: 633
found review approved pr: 628
found review approved pr: 627
found review approved pr: 626
found review approved pr: 625
found review approved pr: 624
found review approved pr: 623
found review approved pr: 622
found review approved pr: 621
found review approved pr: 620
found review approved pr: 616
found review approved pr: 614
found review approved pr: 612
found review approved pr: 610
found review approved pr: 606
found review approved pr: 605
github code reviews found
Contributors: Pass 10
companies found: kubeflow,google,datto,googlers
Frozen-Deps: Pass 10
go modules found: go.mod
Fuzzing: Fail 10
Packaging: Pass 10
!! not a packaging workflow: .github/workflows/codeql-analysis.yml
found docker publishing workflow: .github/workflows/gitcache-docker.yaml
found a completed run: https://github.com/ossf/scorecard/actions/runs/1004287597
Pull-Requests: Pass 10
found commit with PR: dcdb452a9791510ef77e754720dd5603b07044a4
found commit with PR: 3181aba22b6a26404528682ec66478d8d86b95d0
found commit with PR: 581e170db1928a21bc8bac6eac7667e4280c54c1
found commit with PR: aab6c217cc453d0d2a57c9b2b42629af30c6450d
found commit with PR: aeead9468088c83765fab39afed34bb4ada79a27
found commit with PR: 68dc079b79e4e9ea822f947cda183fe96dc80d1f
found commit with PR: c61a744c1ba5619dbfc19ce4257ffbf12056a47a
found commit with PR: e06ce1529d1daf3a38f91d1ae8e13209773eb1d9
found commit with PR: 08e934cbc23f6fd15014dbdb0019855fe1c8b690
found commit with PR: d81fd2424682498c7d2842a8f0904d262ea59ab0
found commit with PR: ecab8fed527efc6a2c49cf3eb0a305341d2ab2e5
found commit with PR: 1c5a247f4411bf0eca9b82abdb74a1784bd4589c
found commit with PR: 2fab8619555bfc83e9ab135fce5ba3dcde2441e7
found commit with PR: fcec1d3a3e914ff8ddef309049ed399e8a142a9f
found commit with PR: 7535a7688cd5072467c8154e6038d473752c9f6f
found commit with PR: dd1a412b857706fdeddbe0cd3027fe33d6ac23e4
found commit with PR: ec7755da826fd027183fe5f716c6decb551ceb99
found commit with PR: 5dd7f118ae9853b5dd58ec27bf6f62ebddb2a39f
found commit with PR: 6a3337d885322e703faa7500ec77588e17c306da
found commit with PR: 2a0031910a884a5fce44d0cc2dd054a360274867
found commit with PR: 6a2a1faa6f04cde06b292204293e687ca4d9bfe2
found commit with PR: fd0bb468367df62bcb4a25d4d80a332026faa0f7
found commit with PR: 18c3178a84117e92e89b63a494fa1199419b5730
found commit with PR: c095d6f161ecefffa2425fbc3040bc24204b1939
found commit with PR: 161d4964bce2f4ca3d0cdb83f0f15c7bd3b43150
found commit with PR: 34621504fbfca26b447f80ba4cd19fe411d09a4f
found commit with PR: 18b53076d6c1f7a016b82fa50a5a6715fe35db1a
found commit with PR: bf87a7a00a467ca51756d47953c5cd5be3c58f3a
found commit with PR: c900290630166887677eef7af3a8c2010e48dd19
found commit with PR: 6aefe1b6ac2a9e2ecd9d7323e6d22f413e4a56eb
found PRs for 30 out of 30 commits
SAST: Pass 10
found CodeQL definition: .github/workflows/codeql-analysis.yml
Security-Policy: Pass 10
security policy : SECURITY.md
Signed-Releases: Fail 3
release found: v2.0.0
signed release artifact found: scorecard_2.0.0_checksums.txt.sig, url: https://api.github.com/repos/ossf/scorecard/releases/assets/39433151
release found: v1.2.0
signed release artifact found: scorecard_1.2.0_checksums.txt.sig, url: https://api.github.com/repos/ossf/scorecard/releases/assets/33614699
release found: v1.1.1
signed release artifact found: scorecard_1.1.1_checksums.txt.sig, url: https://api.github.com/repos/ossf/scorecard/releases/assets/32226313
release found: v1.1.0
!! release v1.1.0 has no signed artifacts
found signed artifacts for 3 out of 4 releases
Signed-Tags: Fail 4
!! unable to find the annotated commit: 87997ffb5724cb479223a08a2890c60b0ea4bfbd
!! unverified tag found: v1.1.0, commit: f2c633854602cf0c8f33164a169fb0a8454bee01, reason: unsigned
verified tag found: v1.1.1, commit: c12de481b2bb26de41f81e905036c6ec4fe739b8
verified tag found: v1.2.0, commit: 30f1eb3da2267135b51fc28c3dfeb0460e2584cb
verified tag found: v2.0.0, commit: bcbe2731d52761e97f40efe740c4ed01cbb48655
found 3 out of 5 verified tags
すごい情報量だ。でも原因がわかる。
Futa Hirakobaどうやら、<指標>: <合否> <スコア>のようだ。
Understanding Scorecard results
Each check returns a Pass / Fail decision, as well as a confidence score between 0 and 10. A confidence of 0 should indicate the check was unable to achieve any real signal, and the result should be ignored. A confidence of 10 indicates the check is completely sure of the result.
https://github.com/ossf/scorecard#understanding-scorecard-results
スコアが10なら最高。
Futa HirakobaActive だけ取ってみる（--checks Active）。それも JSON で（--format json）。
scorecard --format json --checks Active --show-details --repo https://github.com/ossf/scorecard
{"level":"info","ts":1625651403.819372,"caller":"roundtripper/roundtripper.go:65","msg":"using token 0 of total 1."}
{"Repo":"github.com/ossf/scorecard","Date":"2021-07-07","Checks":[{"CheckName":"Active","Pass":true,"Confidence":10,"Details":["commits in last 90 days: 30"]}],"MetaData":[]}
結果
{
"Repo": "github.com/ossf/scorecard",
"Date": "2021-07-07",
"Checks": [
{
"CheckName": "Active",
"Pass": true,
"Confidence": 10,
"Details": [
"commits in last 90 days: 30"
]
}
],
"MetaData": []
}
ちなみに、--show-detailsを入れないと.Checks.Detailsがnullになった。
OSSがActiveに開発されているかどうかを測れそうだ。
Futa Hirakoba我らが assam でも試してみた。
RESULTS
-------
Active: Pass 10
Branch-Protection: Fail 0
CI-Tests: Pass 10
CII-Best-Practices: Fail 10
Code-Review: Pass 10
Contributors: Pass 10
Frozen-Deps: Pass 10
Fuzzing: Fail 10
Packaging: Fail 10
Pull-Requests: Pass 10
SAST: Fail 10
Security-Policy: Fail 0
Signed-Releases: Fail 10
Signed-Tags: Fail 10
10 点で Fail ってなんなんだろ。調べてもよくわからん。
あとちょくちょく404エラーになってるのが気になる
Futa Hirakobaあーなるほどリポジトリの Admin がないといけないのね。
セキュリティポリシーも同じ理由...？それとも SECURITY.md がいる？
詳細な話はscorecard/checks.md at main · ossf/scorecardに載ってた。対策も一緒に。
Futa Hirakobaなーんとなくわかってきたぞ。Pass だからスコアが高いわけでもないっぽい。それぞれのメトリクスの基準をクリアするかどうかで Pass になり、それとは別に信頼性スコアがある。
例えばSigned-ReleasesがFailでもまあ信頼できるということっぽい
このスクラップは2021/07/07にクローズされました