Closed2021/07/07にクローズ11

ossf/scorecard 使ってみる

Security

OSS

Futa Hirakoba 2021/07/07に更新

これ。
https://github.com/ossf/scorecard

最近v2が出た。
Google Online Security Blog: Measuring Security Risks in Open Source Software: Scorecards Launches V2

Futa Hirakoba 2021/07/07

インストール方法が見つからない。go でできてるので go install で入れてみる。

❯ go install github.com/ossf/scorecard@latest

入った。

❯ scorecard version                                 
Error: unknown command "version" for "./scorecard"
Run './scorecard --help' for usage.
unknown command "version" for "./scorecard"

バージョン確認しようとしたがまさかのバージョンはコマンドに含まれてなかった。

Futa Hirakoba 2021/07/07

試しに自信を評価させてみる。

❯ scorecard --repo=https://github.com/ossf/scorecard
Starting [Active]
Starting [Branch-Protection]
Starting [CI-Tests]
Starting [CII-Best-Practices]
Starting [Code-Review]
Starting [Contributors]
Starting [Frozen-Deps]
Starting [Fuzzing]
Starting [Packaging]
Starting [Pull-Requests]
Starting [SAST]
Starting [Security-Policy]
Starting [Signed-Releases]
Starting [Signed-Tags]
{"level":"warn","ts":1625650068.096907,"caller":"roundtripper/roundtripper.go:173","msg":"Rate limit exceeded. Waiting 15m20.903105s to retry..."}
{"level":"warn","ts":1625650068.097593,"caller":"roundtripper/roundtripper.go:173","msg":"Rate limit exceeded. Waiting 15m20.90241s to retry..."}
{"level":"warn","ts":1625650068.097617,"caller":"roundtripper/roundtripper.go:173","msg":"Rate limit exceeded. Waiting 15m20.902415s to retry..."}
{"level":"warn","ts":1625650068.251936,"caller":"roundtripper/roundtripper.go:173","msg":"Rate limit exceeded. Waiting 59m59.7481s to retry..."}
{"level":"warn","ts":1625650068.291726,"caller":"roundtripper/roundtripper.go:173","msg":"Rate limit exceeded. Waiting 15m20.708281s to retry..."}
{"level":"warn","ts":1625650068.3224268,"caller":"roundtripper/roundtripper.go:173","msg":"Rate limit exceeded. Waiting 15m20.677579s to retry..."}
{"level":"warn","ts":1625650068.322602,"caller":"roundtripper/roundtripper.go:173","msg":"Rate limit exceeded. Waiting 15m20.677401s to retry..."}
Finished [Fuzzing]
Finished [SAST]
{"level":"warn","ts":1625650068.3705778,"caller":"roundtripper/roundtripper.go:173","msg":"Rate limit exceeded. Waiting 15m20.629464s to retry..."}
{"level":"warn","ts":1625650068.388341,"caller":"roundtripper/roundtripper.go:173","msg":"Rate limit exceeded. Waiting 15m20.611666s to retry..."}
Finished [Security-Policy]
Finished [Frozen-Deps]
Finished [CII-Best-Practices]
^C

めちゃRate limit exceededで怒られた。
そしてFinished [CII-Best-Practices]から動かなくなった。

Futa Hirakoba 2021/07/07

どうやら GitHub の API を叩きまくるから Personal Access Token で認証しないとすぐ rate limit に到達してしまうらしい。

Before running Scorecard, you need to, either:
create a GitHub access token and set it in environment variable GITHUB_AUTH_TOKEN. This helps to avoid the GitHub's api rate limits with unauthenticated requests.
Authentication

なんの権限も付与していないPATを作成した

Futa Hirakoba 2021/07/07

おーちゃんと動いた。

GITHUB_AUTH_TOKEN=xxxx scorecard --repo=https://github.com/ossf/scorecard
Starting [Active]
Starting [Branch-Protection]
Starting [CI-Tests]
Starting [CII-Best-Practices]
Starting [Code-Review]
Starting [Contributors]
Starting [Frozen-Deps]
Starting [Fuzzing]
Starting [Packaging]
Starting [Pull-Requests]
Starting [SAST]
Starting [Security-Policy]
Starting [Signed-Releases]
Starting [Signed-Tags]
{"level":"info","ts":1625650300.342906,"caller":"roundtripper/roundtripper.go:65","msg":"using token 0 of total 1."}
{"level":"info","ts":1625650300.342999,"caller":"roundtripper/roundtripper.go:65","msg":"using token 0 of total 1."}
{"level":"info","ts":1625650300.3431182,"caller":"roundtripper/roundtripper.go:65","msg":"using token 0 of total 1."}
{"level":"info","ts":1625650300.343125,"caller":"roundtripper/roundtripper.go:65","msg":"using token 0 of total 1."}
{"level":"info","ts":1625650300.343206,"caller":"roundtripper/roundtripper.go:65","msg":"using token 0 of total 1."}
{"level":"info","ts":1625650300.343265,"caller":"roundtripper/roundtripper.go:65","msg":"using token 0 of total 1."}
{"level":"info","ts":1625650300.3433862,"caller":"roundtripper/roundtripper.go:65","msg":"using token 0 of total 1."}
{"level":"info","ts":1625650300.343329,"caller":"roundtripper/roundtripper.go:65","msg":"using token 0 of total 1."}
{"level":"info","ts":1625650300.343426,"caller":"roundtripper/roundtripper.go:65","msg":"using token 0 of total 1."}
{"level":"info","ts":1625650300.342905,"caller":"roundtripper/roundtripper.go:65","msg":"using token 0 of total 1."}
{"level":"info","ts":1625650300.3431401,"caller":"roundtripper/roundtripper.go:65","msg":"using token 0 of total 1."}
{"level":"info","ts":1625650300.3444328,"caller":"roundtripper/roundtripper.go:65","msg":"using token 0 of total 1."}
{"level":"info","ts":1625650300.344446,"caller":"roundtripper/roundtripper.go:65","msg":"using token 0 of total 1."}
{"level":"info","ts":1625650300.344953,"caller":"roundtripper/roundtripper.go:65","msg":"using token 0 of total 1."}
Finished [Fuzzing]
Finished [SAST]
Finished [Security-Policy]
Finished [CII-Best-Practices]
Finished [Frozen-Deps]
Finished [Branch-Protection]
Finished [Signed-Tags]
Finished [Signed-Releases]
Finished [Packaging]
Finished [Contributors]
Finished [Active]
Finished [Pull-Requests]
Finished [Code-Review]
Finished [CI-Tests]

RESULTS
-------
Active: Pass 10
Branch-Protection: Fail 0
CI-Tests: Pass 10
CII-Best-Practices: Fail 10
Code-Review: Pass 10
Contributors: Pass 10
Frozen-Deps: Pass 10
Fuzzing: Fail 10
Packaging: Pass 10
Pull-Requests: Pass 10
SAST: Pass 10
Security-Policy: Pass 10
Signed-Releases: Fail 3
Signed-Tags: Fail 4

Futa Hirakoba 2021/07/07

このままじゃよくわからない。--show-detailsしてみる。

前半は省略

RESULTS
-------
Active: Pass 10
    commits in last 90 days: 30
Branch-Protection: Fail 0
    error, retrying: GET https://api.github.com/repos/ossf/scorecard/branches/main/protection: 404 Not Found []
CI-Tests: Pass 10
    CI test found: pr: 663, context: github-actionsuccess, url: https://api.github.com/repos/ossf/scorecard/check-runs/2998449248uccess
    CI test found: pr: 658, context: github-actionsuccess, url: https://api.github.com/repos/ossf/scorecard/check-runs/2988216073uccess
    CI test found: pr: 656, context: github-actionsuccess, url: https://api.github.com/repos/ossf/scorecard/check-runs/2984995844uccess
    CI test found: pr: 654, context: github-actionsuccess, url: https://api.github.com/repos/ossf/scorecard/check-runs/2985243874uccess
    CI test found: pr: 652, context: github-actionsuccess, url: https://api.github.com/repos/ossf/scorecard/check-runs/2985208338uccess
    CI test found: pr: 649, context: github-actionsuccess, url: https://api.github.com/repos/ossf/scorecard/check-runs/2975804273uccess
    CI test found: pr: 643, context: github-actionsuccess, url: https://api.github.com/repos/ossf/scorecard/check-runs/2970564487uccess
    CI test found: pr: 641, context: github-actionsuccess, url: https://api.github.com/repos/ossf/scorecard/check-runs/2967571364uccess
    CI test found: pr: 640, context: github-actionsuccess, url: https://api.github.com/repos/ossf/scorecard/check-runs/2967020555uccess
    CI test found: pr: 637, context: github-actionsuccess, url: https://api.github.com/repos/ossf/scorecard/check-runs/2959402000uccess
    CI test found: pr: 635, context: github-actionsuccess, url: https://api.github.com/repos/ossf/scorecard/check-runs/2955448364uccess
    CI test found: pr: 634, context: github-actionsuccess, url: https://api.github.com/repos/ossf/scorecard/check-runs/2945495569uccess
    CI test found: pr: 633, context: github-actionsuccess, url: https://api.github.com/repos/ossf/scorecard/check-runs/2945018883uccess
    CI test found: pr: 628, context: github-actionsuccess, url: https://api.github.com/repos/ossf/scorecard/check-runs/2938130596uccess
    CI test found: pr: 627, context: github-actionsuccess, url: https://api.github.com/repos/ossf/scorecard/check-runs/2944692090uccess
    CI test found: pr: 626, context: github-actionsuccess, url: https://api.github.com/repos/ossf/scorecard/check-runs/2938835879uccess
    CI test found: pr: 625, context: github-actionsuccess, url: https://api.github.com/repos/ossf/scorecard/check-runs/2936738100uccess
    CI test found: pr: 624, context: github-actionsuccess, url: https://api.github.com/repos/ossf/scorecard/check-runs/2943644563uccess
    CI test found: pr: 623, context: github-actionsuccess, url: https://api.github.com/repos/ossf/scorecard/check-runs/2944501189uccess
    CI test found: pr: 622, context: github-actionsuccess, url: https://api.github.com/repos/ossf/scorecard/check-runs/2935636381uccess
    CI test found: pr: 621, context: github-actionsuccess, url: https://api.github.com/repos/ossf/scorecard/check-runs/2935956732uccess
    CI test found: pr: 620, context: github-actionsuccess, url: https://api.github.com/repos/ossf/scorecard/check-runs/2928721054uccess
    CI test found: pr: 616, context: github-actionsuccess, url: https://api.github.com/repos/ossf/scorecard/check-runs/2924458994uccess
    CI test found: pr: 614, context: github-actionsuccess, url: https://api.github.com/repos/ossf/scorecard/check-runs/2936436628uccess
    CI test found: pr: 612, context: github-actionsuccess, url: https://api.github.com/repos/ossf/scorecard/check-runs/2909709728uccess
    CI test found: pr: 610, context: github-actionsuccess, url: https://api.github.com/repos/ossf/scorecard/check-runs/2916273401uccess
    CI test found: pr: 606, context: github-actionsuccess, url: https://api.github.com/repos/ossf/scorecard/check-runs/2907147171uccess
    CI test found: pr: 605, context: github-actionsuccess, url: https://api.github.com/repos/ossf/scorecard/check-runs/2899455987uccess
    found CI tests for 28 of 28 merged PRs
CII-Best-Practices: Fail 10
    no badge found
Code-Review: Pass 10
    found review approved pr: 663
    found review approved pr: 658
    found review approved pr: 656
    found review approved pr: 654
    found review approved pr: 652
    found review approved pr: 649
    found review approved pr: 643
    found review approved pr: 641
    found review approved pr: 640
    found review approved pr: 637
    found review approved pr: 635
    found review approved pr: 634
    found review approved pr: 633
    found review approved pr: 628
    found review approved pr: 627
    found review approved pr: 626
    found review approved pr: 625
    found review approved pr: 624
    found review approved pr: 623
    found review approved pr: 622
    found review approved pr: 621
    found review approved pr: 620
    found review approved pr: 616
    found review approved pr: 614
    found review approved pr: 612
    found review approved pr: 610
    found review approved pr: 606
    found review approved pr: 605
    github code reviews found
Contributors: Pass 10
    companies found: kubeflow,google,datto,googlers
Frozen-Deps: Pass 10
    go modules found: go.mod
Fuzzing: Fail 10
Packaging: Pass 10
    !! not a packaging workflow: .github/workflows/codeql-analysis.yml
    found docker publishing workflow: .github/workflows/gitcache-docker.yaml
    found a completed run: https://github.com/ossf/scorecard/actions/runs/1004287597
Pull-Requests: Pass 10
    found commit with PR: dcdb452a9791510ef77e754720dd5603b07044a4
    found commit with PR: 3181aba22b6a26404528682ec66478d8d86b95d0
    found commit with PR: 581e170db1928a21bc8bac6eac7667e4280c54c1
    found commit with PR: aab6c217cc453d0d2a57c9b2b42629af30c6450d
    found commit with PR: aeead9468088c83765fab39afed34bb4ada79a27
    found commit with PR: 68dc079b79e4e9ea822f947cda183fe96dc80d1f
    found commit with PR: c61a744c1ba5619dbfc19ce4257ffbf12056a47a
    found commit with PR: e06ce1529d1daf3a38f91d1ae8e13209773eb1d9
    found commit with PR: 08e934cbc23f6fd15014dbdb0019855fe1c8b690
    found commit with PR: d81fd2424682498c7d2842a8f0904d262ea59ab0
    found commit with PR: ecab8fed527efc6a2c49cf3eb0a305341d2ab2e5
    found commit with PR: 1c5a247f4411bf0eca9b82abdb74a1784bd4589c
    found commit with PR: 2fab8619555bfc83e9ab135fce5ba3dcde2441e7
    found commit with PR: fcec1d3a3e914ff8ddef309049ed399e8a142a9f
    found commit with PR: 7535a7688cd5072467c8154e6038d473752c9f6f
    found commit with PR: dd1a412b857706fdeddbe0cd3027fe33d6ac23e4
    found commit with PR: ec7755da826fd027183fe5f716c6decb551ceb99
    found commit with PR: 5dd7f118ae9853b5dd58ec27bf6f62ebddb2a39f
    found commit with PR: 6a3337d885322e703faa7500ec77588e17c306da
    found commit with PR: 2a0031910a884a5fce44d0cc2dd054a360274867
    found commit with PR: 6a2a1faa6f04cde06b292204293e687ca4d9bfe2
    found commit with PR: fd0bb468367df62bcb4a25d4d80a332026faa0f7
    found commit with PR: 18c3178a84117e92e89b63a494fa1199419b5730
    found commit with PR: c095d6f161ecefffa2425fbc3040bc24204b1939
    found commit with PR: 161d4964bce2f4ca3d0cdb83f0f15c7bd3b43150
    found commit with PR: 34621504fbfca26b447f80ba4cd19fe411d09a4f
    found commit with PR: 18b53076d6c1f7a016b82fa50a5a6715fe35db1a
    found commit with PR: bf87a7a00a467ca51756d47953c5cd5be3c58f3a
    found commit with PR: c900290630166887677eef7af3a8c2010e48dd19
    found commit with PR: 6aefe1b6ac2a9e2ecd9d7323e6d22f413e4a56eb
    found PRs for 30 out of 30 commits
SAST: Pass 10
    found CodeQL definition: .github/workflows/codeql-analysis.yml
Security-Policy: Pass 10
    security policy : SECURITY.md
Signed-Releases: Fail 3
    release found: v2.0.0
    signed release artifact found: scorecard_2.0.0_checksums.txt.sig, url: https://api.github.com/repos/ossf/scorecard/releases/assets/39433151
    release found: v1.2.0
    signed release artifact found: scorecard_1.2.0_checksums.txt.sig, url: https://api.github.com/repos/ossf/scorecard/releases/assets/33614699
    release found: v1.1.1
    signed release artifact found: scorecard_1.1.1_checksums.txt.sig, url: https://api.github.com/repos/ossf/scorecard/releases/assets/32226313
    release found: v1.1.0
    !! release v1.1.0 has no signed artifacts
    found signed artifacts for 3 out of 4 releases
Signed-Tags: Fail 4
    !! unable to find the annotated commit: 87997ffb5724cb479223a08a2890c60b0ea4bfbd
    !! unverified tag found: v1.1.0, commit: f2c633854602cf0c8f33164a169fb0a8454bee01, reason: unsigned
    verified tag found: v1.1.1, commit: c12de481b2bb26de41f81e905036c6ec4fe739b8
    verified tag found: v1.2.0, commit: 30f1eb3da2267135b51fc28c3dfeb0460e2584cb
    verified tag found: v2.0.0, commit: bcbe2731d52761e97f40efe740c4ed01cbb48655
    found 3 out of 5 verified tags

すごい情報量だ。でも原因がわかる。

Futa Hirakoba 2021/07/07

どうやら、<指標>: <合否> <スコア>のようだ。

Understanding Scorecard results

Each check returns a Pass / Fail decision, as well as a confidence score between 0 and 10. A confidence of 0 should indicate the check was unable to achieve any real signal, and the result should be ignored. A confidence of 10 indicates the check is completely sure of the result.
https://github.com/ossf/scorecard#understanding-scorecard-results

スコアが10なら最高。

Futa Hirakoba 2021/07/07

Active だけ取ってみる（--checks Active）。それも JSON で（--format json）。

scorecard --format json --checks Active --show-details --repo https://github.com/ossf/scorecard
{"level":"info","ts":1625651403.819372,"caller":"roundtripper/roundtripper.go:65","msg":"using token 0 of total 1."}
{"Repo":"github.com/ossf/scorecard","Date":"2021-07-07","Checks":[{"CheckName":"Active","Pass":true,"Confidence":10,"Details":["commits in last 90 days: 30"]}],"MetaData":[]}

結果

{
  "Repo": "github.com/ossf/scorecard",  
  "Date": "2021-07-07",
  "Checks": [
    {
      "CheckName": "Active",
      "Pass": true,
      "Confidence": 10,
      "Details": [
        "commits in last 90 days: 30"
      ]
    }
  ],
  "MetaData": []
}

ちなみに、--show-detailsを入れないと.Checks.Detailsがnullになった。
OSSがActiveに開発されているかどうかを測れそうだ。

Futa Hirakoba 2021/07/07

我らが assam でも試してみた。

RESULTS
-------
Active: Pass 10
Branch-Protection: Fail 0
CI-Tests: Pass 10
CII-Best-Practices: Fail 10
Code-Review: Pass 10
Contributors: Pass 10
Frozen-Deps: Pass 10
Fuzzing: Fail 10
Packaging: Fail 10
Pull-Requests: Pass 10
SAST: Fail 10
Security-Policy: Fail 0
Signed-Releases: Fail 10
Signed-Tags: Fail 10

10 点で Fail ってなんなんだろ。調べてもよくわからん。
あとちょくちょく404エラーになってるのが気になる

Futa Hirakoba 2021/07/07

あーなるほどリポジトリの Admin がないといけないのね。

セキュリティポリシーも同じ理由...？それとも SECURITY.md がいる？
詳細な話はscorecard/checks.md at main · ossf/scorecardに載ってた。対策も一緒に。

Futa Hirakoba 2021/07/07

なーんとなくわかってきたぞ。Pass だからスコアが高いわけでもないっぽい。それぞれのメトリクスの基準をクリアするかどうかで Pass になり、それとは別に信頼性スコアがある。

例えばSigned-ReleasesがFailでもまあ信頼できるということっぽい

このスクラップは2021/07/07にクローズされました