⛈️

CodeBuildのローカルビルド実行時の「ERROR: client version 1.22 is too old.… 」への対処方法

2024/02/15に公開

AWS

TL;DR

codebuild/local-buildsの/LocalBuild/agent-resources/docker-compose.ymlのversionを3以上に上げる
docker-compose.yml内のBMR:3000をagent:3000に変更する
docker-compose.yml内のlinksを削除する

背景

ローカル環境でCodeBuildを実行しようとしたところ、codebuild_build.shが呼び出されてすぐに以下のようなエラーとなり終了してしまう。

ERROR: client version 1.22 is too old. Minimum supported API version is 1.24, please upgrade your client to a newer version
ERROR: client version 1.22 is too old. Minimum supported API version is 1.24, please upgrade your client to a newer version

対処方法

修正版のlocal-buildsイメージを使用する。

以下のようなDockerfileを作成

Dockerfile

FROM public.ecr.aws/codebuild/local-builds:latest

COPY "./docker-compose.yml" \
    "/LocalBuild/agent-resources/docker-compose.yml"
COPY "./docker-compose-mount-src-dir.yml" \
    "/LocalBuild/agent-resources/docker-compose-mount-src-dir.yml"

codebuild_build.shを同じ階層に置いた状態で以下実行

command

export LOCALBUILD_CUSTOM_IMAGE_TAG="my-local-builds:latest" && \
    export LOCALBUILD_WORK_DIR="/workdir" && \
    docker run \
        -it \
        -v ./:"${LOCALBUILD_WORK_DIR}" \
        --rm \
        --entrypoint=/bin/bash \
        public.ecr.aws/codebuild/local-builds:latest \
            -c "curl \
                -L \
                -o /usr/bin/yq \
                https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64 2>/dev/null&& \
                chmod +x /usr/bin/yq && \
                cp /LocalBuild/agent-resources/{docker-compose.yml,docker-compose-mount-src-dir.yml} ${LOCALBUILD_WORK_DIR} && \
                for file in \"docker-compose.yml\" \"docker-compose-mount-src-dir.yml\"; do \
                    yq -i \
                        '.version=\"3\" | \
                        .services.build.environment[0] = \"NO_PROXY=agent:3000\" |
                        .services.build.environment[2] = \"CODEBUILD_AGENT_PORT=http://agent:3000\" |
                        del(.services.build.links)' \
                        ${LOCALBUILD_WORK_DIR}/\${file}; \
                done" && \
    docker build -t "${LOCALBUILD_CUSTOM_IMAGE_TAG}" . && \
    sed -i "s/public\.ecr\.aws\/codebuild\/local-builds:latest/${LOCALBUILD_CUSTOM_IMAGE_TAG}/g" codebuild_build.sh

※local-buildsはmy-local-buildsというカスタムイメージに置き換えて実行されるようになる。

解析内容

docker-compose.ymlのバージョン

エラーを調べた限りではどの記事でもdocker-compose.ymlのバージョンが古いとのことだった。
しかし、codebuild_build.shで最初に起動されるコンテナpublic.ecr.aws/codebuild/local-buildsについてはソースコードが公開されていないうえ、細かい動作仕様についても公開されていないため、手探りで推測しながら解析することとした。

local-builds起動時に呼ばれる/usr/local/bin/local_build.shを見たところ、/LocalBuild/agent-resourcesにあるdocker-compose.yml,docker-compose-mount-src-dir.ymlをベースに修正したymlファイルが実行コンテナのdocker-composeに使われているようである。

/usr/local/bin/local_build.sh

if [ "${MOUNT_SOURCE_DIRECTORY}" != "TRUE" ]
then
    cp docker-compose.yml customer-specific.yml
    export CODEBUILD_MOUNT_SRC_DIR=false
else
    cp docker-compose-mount-src-dir.yml customer-specific.yml
    export CODEBUILD_MOUNT_SRC_DIR=true
fi

ここで、docker-compose.ymlを見てみるとversionが2となっていた。

/LocalBuild/agent-resources/docker-compose.yml

version: '2'

services:
  agent:

解決策の通り、versionの値を上げてcodebuild_build.shで修正したファイルをマウントするようにした。※2.1でも良いようだが低すぎるといずれ再発するのでひとまず3としている。

docker-compose.yml

version: '3'

services:
  agent:

codebuild_build.sh

if isOSWindows
then
    docker_command+=" -e \"INITIATOR=$USERNAME\""
else
    docker_command+=" -e \"INITIATOR=$USER\""
fi

docker_command+=" -v ./docker-compose.yml:/LocalBuild/agent-resources/docker-compose.yml"

if [ -n "$local_agent_image" ]
then
    docker_command+=" $local_agent_image"
else
    docker_command+=" public.ecr.aws/codebuild/local-builds:latest"
fi

Waiting for agent pingから進まない

再度codebuild_build.shを実行すると、上記エラーは解消したが、以下のログで止まってしまうようになった。

Creating agent-resources_agent_1 ... done
Creating agent-resources_build_1 ... done
Attaching to agent-resources_agent_1, agent-resources_build_1
agent_1  | [Container] ****/**/** **:**:** Waiting for agent ping

※docker-compose.ymlのversionを3.7等にしても変化なし
このログはlocal_build.shで呼ばれる/LocalBuild/agent-resources/start内のtouch $EXECUTOR_DIR/executor.doneが実行されてから出力されていて、ビルド側コンテナの/codebuild/readonly/bin/executorが出しているようにみえた。

/LocalBuild/agent-resources/start

EXECUTOR_DIR=/codebuild/input/bin
mkdir -p $EXECUTOR_DIR
cp bin/executor $EXECUTOR_DIR/
touch $EXECUTOR_DIR/executor.done

/docker-compose.yml

build:
  entrypoint: sh -c "while [ ! -f /codebuild/readonly/bin/executor.done ]; do sleep 1; done && /codebuild/readonly/bin/executor > /dev/null"

また、そのあとagent側で呼ばれる/LocalBuild/agent-resources/bin/agentのプロセスが終わるまでagent側コンテナは待ち続けているようなので、この間に何らかの問題が起きていると推測した。

/LocalBuild/agent-resources/start

# Used by logger
hostname=$(hostname -I)
hostname_arr=($hostname)
export CODEBUILD_AGENT_ENDPOINT="http://${hostname_arr[0]}:3000"
echo "$CODEBUILD_AGENT_ENDPOINT" > $EXECUTOR_DIR/ip_address.txt

if [ "$CODEBUILD_GOLANG_AGENT" != "" ]; then
  ./bin/agent -port=3000 & echo $! > ./build-agent.pid
fi
・・・
wait $(cat ./build-agent.pid)

このagentファイルはGo言語で実装されたデバッグ情報ほぼ無し、ソースコードも公開なしのバイナリであった。
そこでagentが呼ばれる直前で無限ループさせ、コンテナにターミナルで入り直接agentを実行してみる。

./agent -port=3000
Starting program: /LocalBuild/agent-resources/bin/agent -port=3000
・・・
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: localhost:8100
・・・
{"buildArn":""}

localhost:8100に対して何かを要求しているようだが、これはdocker-compose.ymlでCODEBUILD_CTS_ENDPOINTという環境変数でURLが定義されていた。

docker-compose.yml

- CODEBUILD_CTS_ENDPOINT=http://localhost:8100

CTSが何者かは全く情報がなかったのでわからなかったが、/LocalBuild/agent-resources/binに同名のバイナリファイルがあったので、試しにこれを別ターミナルで起動した状態でagentを再度実行してみる。

./cts
-----
{"agentFeatures":["ALLOW_NATIVE_GIT","NATIVE_GIT_GITHUB","NATIVE_GIT_GHE","NATIVE_GIT_BITBUCKET","NATIVE_GIT_CODECOMMIT"],"buildArn":""}
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
・・・
{"buildArn":"","metrics":[{"dimensions":[{"instance":"PROVISIONING","metricClass":"PhaseType"}],"metricValue":0,"name":"PhaseFailed","type":"COUNT"},{"dimensions":[{"instance":"PROVISIONING","metricClass":"PhaseType"}],"metricValue":0,"name":"PhaseFault","type":"COUNT"},{"dimensions":[{"instance":"PROVISIONING","metricClass":"PhaseType"}],"metricValue":1,"name":"PhaseSucceeded","type":"COUNT"},{"metricValue":0,"name":"EFS","type":"COUNT"}]}
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
・・・

何かの処理がやり取りされているように見えるが、最後のレスポンスを境に処理が止まってしまった。
また、cts側を手動で起動しても何もログが出てこず手掛かりを得ることができなかった。
ctsは一部デバッグ情報があるようだったがそもそもソースコードが入手できていないため、これらの実行ファイルを用いた解析は困難と判断して中断した。

ビルド側コンテナでの名前解決失敗

ここで、別の手掛かりとして引数や周辺にポート3000を指定していたことと、ログからpingで止まっていると思われることより、どこかでネットワーク疎通がうまくいっていないのではと推測した。
上記docker-compose.ymlで、buildのコンテナ側で設定されていた環境変数の中に3000ポートを使用している箇所が数か所あった。

docker-compose.yml

environment:
  - NO_PROXY=BMR:3000
  - CODEBUILD_LOCAL_BUILD=true
  - CODEBUILD_AGENT_PORT=http://BMR:3000

そのうえでagentは3000ポートをEXPOSEしていたので、おそらくbuild⇔agent間のやり取りはagent側のホスト名をBMRとし、3000ポートで実行しているようにみえる。

docker-compose.yml

agent:
  image: ${LOCAL_AGENT_IMAGE}
  ports:
    - "3000"

試しにWaiting for agent pingで止まっている最中にビルド側コンテナに入り、この「BMR」へpingが通るかを確認した。

ping BMR
ping: BMR: Name or service not known

このdocker-compose.ymlではbuildでlinksによる設定でagentとBMRを対応付けているようだが、ホスト名が解決できずにpingの疎通できていないところを見るとおそらくlinksの設定がホスト側のDocker環境によっては機能していないように思われる。

docker-compose.yml

      - CODEBUILD_INITIATOR=${CODEBUILD_LOCAL_INITIATOR}
    links:
      - "agent:BMR"
volumes:

そのうえで、処理の内容を見る限りはビルド側のコンテナからエージェント側のコンテナにBMR:3000に対して何かアクセスすることでビルド処理が始まるのではと推測し、以下のような修正を行って再度確認した。

BMR:3000を「agent:3000」に置換
linksを削除

docker-compose.yml

    environment:
      - NO_PROXY=agent:3000
      - CODEBUILD_LOCAL_BUILD=true
      - CODEBUILD_AGENT_PORT=http://agent:3000
      - CODEBUILD_AUTH_TOKEN=12345
      - CODEBUILD_BUILD_ID=local:00000000-0000-0000-0000-000000000000
      - CODEBUILD_BUILD_IMAGE=${IMAGE_FOR_CODEBUILD_LOCAL_BUILD}
      - CODEBUILD_INITIATOR=${CODEBUILD_LOCAL_INITIATOR}
volumes:

修正後の実行結果

Creating agent-resources_build_1 ... done
Creating agent-resources_agent_1 ... done
Attaching to agent-resources_agent_1, agent-resources_build_1
agent_1  | [Container] ****/**/** **:**:** Waiting for agent ping
agent_1  | [Container] ****/**/** **:**:** Waiting for DOWNLOAD_SOURCE
agent_1  | [Container] ****/**/** **:**:** Phase is DOWNLOAD_SOURCE
・・・
agent_1  | [Container] ****/**/** **:**:** Entering phase BUILD
agent_1  | [Container] ****/**/** **:**:** Running command echo "Build test"
agent_1  | Build test
agent_1  | 
agent_1  | [Container] ****/**/** **:**:** Phase complete: BUILD State: SUCCEEDED
・・・
agent-resources_agent_1 exited with code 0

Waiting for agent pingで止まることなくbuildspec.ymlでのコマンドが期待通り実行されるようになった。
以上より、ビルド側コンテナ→agentコンテナの名前解決の失敗によることが直接的な原因とみてほぼほぼ間違いないと考える。

これらの修正をまとめてコマンド化・カスタムイメージ化したのが対処方法の通りである。

余談

修正後のイメージに対して再度ctsとagentを手動実行すると以下のように結果が変化した。
この間にビルド側コンテナでビルドが進んだので、agentが3000ポートで待機してビルドコンテナからの何らかの要求を受け付けてctsともやり取りをしながら状態を変化させてビルドを進ませていると思われる。

{"agentFeatures":["ALLOW_NATIVE_GIT","NATIVE_GIT_GITHUB","NATIVE_GIT_GHE","NATIVE_GIT_BITBUCKET","NATIVE_GIT_CODECOMMIT"],"buildArn":""}
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
・・・
-----------------------------------------------------
****/**/** **:**:** {"agentCredentials":{"awsAccessKey":"mock-aws-access-key","awsSecretKey":"mock-aws-secret-key","sessionToken":"mock-aws-session-token"},"outputArtifacts":{"outputArtifacts":{"packaging":"ZIP","type":"LOCAL"}},"secondarySources":[],"sourceLocation":{"location":"","sourceIdentifier":"","type":"LOCAL"}}
・・・
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: localhost:8100
・・・
{"agentFeatures":["ROCOCO_SNS_NOTIFICATION"],"buildArn":""}
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
・・・
-----------------------------------------------------
****/**/** **:**:** {"agentCredentials":{"awsAccessKey":"mock-aws-access-key","awsSecretKey":"mock-aws-secret-key","sessionToken":"mock-aws-session-token"},"outputArtifacts":{"outputArtifacts":{"packaging":"ZIP","type":"LOCAL"}},"secondarySources":[],"sourceLocation":{"location":"","sourceIdentifier":"","type":"LOCAL"}}
・・・
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: localhost:8100
・・・
{"buildArn":""}
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
・・・
-----------------------------------------------------
****/**/** **:**:** {"agentCredentials":{"awsAccessKey":"mock-aws-access-key","awsSecretKey":"mock-aws-secret-key","sessionToken":"mock-aws-session-token"},"outputArtifacts":{"outputArtifacts":{"packaging":"ZIP","type":"LOCAL"}},"secondarySources":[],"sourceLocation":{"location":"","sourceIdentifier":"","type":"LOCAL"}}
・・・
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: localhost:8100
・・・
{"agentFeatures":["CAWS_BUILD"],"buildArn":""}
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
・・・
-----------------------------------------------------
****/**/** **:**:** {"agentCredentials":{"awsAccessKey":"mock-aws-access-key","awsSecretKey":"mock-aws-secret-key","sessionToken":"mock-aws-session-token"},"outputArtifacts":{"outputArtifacts":{"packaging":"ZIP","type":"LOCAL"}},"secondarySources":[],"sourceLocation":{"location":"","sourceIdentifier":"","type":"LOCAL"}}
・・・
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: localhost:8100
・・・
{"agentFeatures":["ON_FAILURE_KEY_BUILDSPEC"],"buildArn":""}
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
・・・
-----------------------------------------------------
****/**/** **:**:** {"agentCredentials":{"awsAccessKey":"mock-aws-access-key","awsSecretKey":"mock-aws-secret-key","sessionToken":"mock-aws-session-token"},"outputArtifacts":{"outputArtifacts":{"packaging":"ZIP","type":"LOCAL"}},"secondarySources":[],"sourceLocation":{"location":"","sourceIdentifier":"","type":"LOCAL"}}
・・・
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
・・・
{"buildArn":"","completedPhaseState":{"contexts":[{"containerType":"build_agent","message":"","statusCode":""}],"outputs":[],"phase":"PROVISIONING","state":"SUCCEEDED"}}
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
・・・
-----------------------------------------------------
****/**/** **:**:** {"nextPhase":"DOWNLOAD_SOURCE"}
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: localhost:8100
・・・
{"agentFeatures":["ALLOW_NATIVE_GIT","NATIVE_GIT_GITHUB","NATIVE_GIT_GHE","NATIVE_GIT_BITBUCKET","NATIVE_GIT_CODECOMMIT"],"buildArn":""}
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
・・・
-----------------------------------------------------
****/**/** **:**:** {"agentCredentials":{"awsAccessKey":"mock-aws-access-key","awsSecretKey":"mock-aws-secret-key","sessionToken":"mock-aws-session-token"},"outputArtifacts":{"outputArtifacts":{"packaging":"ZIP","type":"LOCAL"}},"secondarySources":[],"sourceLocation":{"location":"","sourceIdentifier":"","type":"LOCAL"}}
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: localhost:8100
・・・
{"agentFeatures":["CAWS_BUILD"],"buildArn":""}
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
・・・
-----------------------------------------------------
****/**/** **:**:** {"agentCredentials":{"awsAccessKey":"mock-aws-access-key","awsSecretKey":"mock-aws-secret-key","sessionToken":"mock-aws-session-token"},"outputArtifacts":{"outputArtifacts":{"packaging":"ZIP","type":"LOCAL"}},"secondarySources":[],"sourceLocation":{"location":"","sourceIdentifier":"","type":"LOCAL"}}
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: localhost:8100
・・・
{"agentFeatures":["ALLOW_NATIVE_GIT","NATIVE_GIT_GITHUB","NATIVE_GIT_GHE","NATIVE_GIT_BITBUCKET","NATIVE_GIT_CODECOMMIT"],"buildArn":""}
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
・・・
-----------------------------------------------------
****/**/** **:**:** {"agentCredentials":{"awsAccessKey":"mock-aws-access-key","awsSecretKey":"mock-aws-secret-key","sessionToken":"mock-aws-session-token"},"outputArtifacts":{"outputArtifacts":{"packaging":"ZIP","type":"LOCAL"}},"secondarySources":[],"sourceLocation":{"location":"","sourceIdentifier":"","type":"LOCAL"}}
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: localhost:8100
・・・
{"branches":[],"buildArn":""}
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
・・・
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: localhost:8100
・・・
{"agentFeatures":["ROCOCO_SNS_NOTIFICATION"],"buildArn":""}
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
・・・
-----------------------------------------------------
****/**/** **:**:** {"agentCredentials":{"awsAccessKey":"mock-aws-access-key","awsSecretKey":"mock-aws-secret-key","sessionToken":"mock-aws-session-token"},"outputArtifacts":{"outputArtifacts":{"packaging":"ZIP","type":"LOCAL"}},"secondarySources":[],"sourceLocation":{"location":"","sourceIdentifier":"","type":"LOCAL"}}
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: localhost:8100
・・・
{"agentFeatures":["ROCOCO_SNS_NOTIFICATION"],"buildArn":""}
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
・・・
****/**/** **:**:** {"agentCredentials":{"awsAccessKey":"mock-aws-access-key","awsSecretKey":"mock-aws-secret-key","sessionToken":"mock-aws-session-token"},"outputArtifacts":{"outputArtifacts":{"packaging":"ZIP","type":"LOCAL"}},"secondarySources":[],"sourceLocation":{"location":"","sourceIdentifier":"","type":"LOCAL"}}
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: localhost:8100
・・・
{"agentFeatures":["ON_FAILURE_KEY_BUILDSPEC"],"buildArn":""}
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
・・・
-----------------------------------------------------
****/**/** **:**:** {"agentCredentials":{"awsAccessKey":"mock-aws-access-key","awsSecretKey":"mock-aws-secret-key","sessionToken":"mock-aws-session-token"},"outputArtifacts":{"outputArtifacts":{"packaging":"ZIP","type":"LOCAL"}},"secondarySources":[],"sourceLocation":{"location":"","sourceIdentifier":"","type":"LOCAL"}}
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: localhost:8100
・・・
{"buildArn":"","completedPhaseState":{"contexts":[{"containerType":"build_agent","message":"","statusCode":""}],"outputs":[],"phase":"DOWNLOAD_SOURCE","state":"SUCCEEDED"}}
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
・・・
-----------------------------------------------------
****/**/** **:**:** {"nextPhase":"INSTALL"}
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: localhost:8100
・・・
{"agentFeatures":["ROCOCO_SNS_NOTIFICATION"],"buildArn":""}
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
・・・
-----------------------------------------------------
****/**/** **:**:** {"agentCredentials":{"awsAccessKey":"mock-aws-access-key","awsSecretKey":"mock-aws-secret-key","sessionToken":"mock-aws-session-token"},"outputArtifacts":{"outputArtifacts":{"packaging":"ZIP","type":"LOCAL"}},"secondarySources":[],"sourceLocation":{"location":"","sourceIdentifier":"","type":"LOCAL"}}
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: localhost:8100
・・・
{"agentFeatures":["ROCOCO_SNS_NOTIFICATION"],"buildArn":""}
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
・・・
-----------------------------------------------------
****/**/** **:**:** {"agentCredentials":{"awsAccessKey":"mock-aws-access-key","awsSecretKey":"mock-aws-secret-key","sessionToken":"mock-aws-session-token"},"outputArtifacts":{"outputArtifacts":{"packaging":"ZIP","type":"LOCAL"}},"secondarySources":[],"sourceLocation":{"location":"","sourceIdentifier":"","type":"LOCAL"}}
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: localhost:8100
・・・
{"agentFeatures":["ON_FAILURE_KEY_BUILDSPEC"],"buildArn":""}
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
・・・
-----------------------------------------------------
****/**/** **:**:** {"agentCredentials":{"awsAccessKey":"mock-aws-access-key","awsSecretKey":"mock-aws-secret-key","sessionToken":"mock-aws-session-token"},"outputArtifacts":{"outputArtifacts":{"packaging":"ZIP","type":"LOCAL"}},"secondarySources":[],"sourceLocation":{"location":"","sourceIdentifier":"","type":"LOCAL"}}
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: localhost:8100
・・・
{"buildArn":"","completedPhaseState":{"contexts":[{"containerType":"build_agent","message":"","statusCode":""}],"outputs":[],"phase":"INSTALL","state":"SUCCEEDED"}}
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
・・・
-----------------------------------------------------
****/**/** **:**:** {"nextPhase":"PRE_BUILD"}
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: localhost:8100
・・・
{"agentFeatures":["ROCOCO_SNS_NOTIFICATION"],"buildArn":""}
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
・・・
-----------------------------------------------------
****/**/** **:**:** {"agentCredentials":{"awsAccessKey":"mock-aws-access-key","awsSecretKey":"mock-aws-secret-key","sessionToken":"mock-aws-session-token"},"outputArtifacts":{"outputArtifacts":{"packaging":"ZIP","type":"LOCAL"}},"secondarySources":[],"sourceLocation":{"location":"","sourceIdentifier":"","type":"LOCAL"}}
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: localhost:8100
・・・
{"agentFeatures":["ROCOCO_SNS_NOTIFICATION"],"buildArn":""}
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
・・・
-----------------------------------------------------
****/**/** **:**:** {"agentCredentials":{"awsAccessKey":"mock-aws-access-key","awsSecretKey":"mock-aws-secret-key","sessionToken":"mock-aws-session-token"},"outputArtifacts":{"outputArtifacts":{"packaging":"ZIP","type":"LOCAL"}},"secondarySources":[],"sourceLocation":{"location":"","sourceIdentifier":"","type":"LOCAL"}}
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: localhost:8100
・・・
{"agentFeatures":["ON_FAILURE_KEY_BUILDSPEC"],"buildArn":""}
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
・・・
-----------------------------------------------------
****/**/** **:**:** {"agentCredentials":{"awsAccessKey":"mock-aws-access-key","awsSecretKey":"mock-aws-secret-key","sessionToken":"mock-aws-session-token"},"outputArtifacts":{"outputArtifacts":{"packaging":"ZIP","type":"LOCAL"}},"secondarySources":[],"sourceLocation":{"location":"","sourceIdentifier":"","type":"LOCAL"}}
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: localhost:8100
・・・
{"buildArn":"","completedPhaseState":{"contexts":[{"containerType":"build_agent","message":"","statusCode":""}],"outputs":[],"phase":"PRE_BUILD","state":"SUCCEEDED"}}
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
・・・
-----------------------------------------------------
****/**/** **:**:** {"nextPhase":"BUILD"}
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: localhost:8100
・・・
{"agentFeatures":["ROCOCO_SNS_NOTIFICATION"],"buildArn":""}
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
・・・
-----------------------------------------------------
****/**/** **:**:** {"agentCredentials":{"awsAccessKey":"mock-aws-access-key","awsSecretKey":"mock-aws-secret-key","sessionToken":"mock-aws-session-token"},"outputArtifacts":{"outputArtifacts":{"packaging":"ZIP","type":"LOCAL"}},"secondarySources":[],"sourceLocation":{"location":"","sourceIdentifier":"","type":"LOCAL"}}
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: localhost:8100
・・・
{"agentFeatures":["ROCOCO_SNS_NOTIFICATION"],"buildArn":""}
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
・・・
-----------------------------------------------------
****/**/** **:**:** {"agentCredentials":{"awsAccessKey":"mock-aws-access-key","awsSecretKey":"mock-aws-secret-key","sessionToken":"mock-aws-session-token"},"outputArtifacts":{"outputArtifacts":{"packaging":"ZIP","type":"LOCAL"}},"secondarySources":[],"sourceLocation":{"location":"","sourceIdentifier":"","type":"LOCAL"}}
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: localhost:8100
・・・
{"agentFeatures":["ON_FAILURE_KEY_BUILDSPEC"],"buildArn":""}
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
・・・
-----------------------------------------------------
****/**/** **:**:** {"agentCredentials":{"awsAccessKey":"mock-aws-access-key","awsSecretKey":"mock-aws-secret-key","sessionToken":"mock-aws-session-token"},"outputArtifacts":{"outputArtifacts":{"packaging":"ZIP","type":"LOCAL"}},"secondarySources":[],"sourceLocation":{"location":"","sourceIdentifier":"","type":"LOCAL"}}
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: localhost:8100
・・・
{"buildArn":"","completedPhaseState":{"contexts":[{"containerType":"build_agent","message":"","statusCode":""}],"outputs":[],"phase":"BUILD","state":"SUCCEEDED"}}
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
・・・
-----------------------------------------------------
****/**/** **:**:** {"nextPhase":"POST_BUILD"}
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: localhost:8100
・・・
{"agentFeatures":["ROCOCO_SNS_NOTIFICATION"],"buildArn":""}
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
・・・
-----------------------------------------------------
****/**/** **:**:** {"agentCredentials":{"awsAccessKey":"mock-aws-access-key","awsSecretKey":"mock-aws-secret-key","sessionToken":"mock-aws-session-token"},"outputArtifacts":{"outputArtifacts":{"packaging":"ZIP","type":"LOCAL"}},"secondarySources":[],"sourceLocation":{"location":"","sourceIdentifier":"","type":"LOCAL"}}
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: localhost:8100
・・・
{"agentFeatures":["ROCOCO_SNS_NOTIFICATION"],"buildArn":""}
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
・・・
-----------------------------------------------------
****/**/** **:**:** {"agentCredentials":{"awsAccessKey":"mock-aws-access-key","awsSecretKey":"mock-aws-secret-key","sessionToken":"mock-aws-session-token"},"outputArtifacts":{"outputArtifacts":{"packaging":"ZIP","type":"LOCAL"}},"secondarySources":[],"sourceLocation":{"location":"","sourceIdentifier":"","type":"LOCAL"}}
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: localhost:8100
・・・
{"agentFeatures":["ON_FAILURE_KEY_BUILDSPEC"],"buildArn":""}
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
・・・
-----------------------------------------------------
****/**/** **:**:** {"agentCredentials":{"awsAccessKey":"mock-aws-access-key","awsSecretKey":"mock-aws-secret-key","sessionToken":"mock-aws-session-token"},"outputArtifacts":{"outputArtifacts":{"packaging":"ZIP","type":"LOCAL"}},"secondarySources":[],"sourceLocation":{"location":"","sourceIdentifier":"","type":"LOCAL"}}
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: localhost:8100
・・・
{"buildArn":"","completedPhaseState":{"contexts":[{"containerType":"build_agent","message":"","statusCode":""}],"outputs":[],"phase":"POST_BUILD","state":"SUCCEEDED"}}
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
・・・
-----------------------------------------------------
****/**/** **:**:** {"nextPhase":"UPLOAD_ARTIFACTS"}
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: localhost:8100
・・・
{"agentFeatures":["ON_FAILURE_KEY_BUILDSPEC"],"buildArn":""}
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
・・・
-----------------------------------------------------
****/**/** **:**:** {"agentCredentials":{"awsAccessKey":"mock-aws-access-key","awsSecretKey":"mock-aws-secret-key","sessionToken":"mock-aws-session-token"},"outputArtifacts":{"outputArtifacts":{"packaging":"ZIP","type":"LOCAL"}},"secondarySources":[],"sourceLocation":{"location":"","sourceIdentifier":"","type":"LOCAL"}}
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: localhost:8100
・・・
{"buildArn":"","completedPhaseState":{"contexts":[{"containerType":"build_agent","message":"","statusCode":""}],"outputs":[],"phase":"UPLOAD_ARTIFACTS","state":"SUCCEEDED"}}
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
・・・
-----------------------------------------------------
****/**/** **:**:** {"nextPhase":"FINALIZING"}
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: localhost:8100
・・・
{"agentFeatures":["ALLOW_NATIVE_GIT","NATIVE_GIT_GITHUB","NATIVE_GIT_GHE","NATIVE_GIT_BITBUCKET","NATIVE_GIT_CODECOMMIT"],"buildArn":""}
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
・・・
-----------------------------------------------------
****/**/** **:**:** {"agentCredentials":{"awsAccessKey":"mock-aws-access-key","awsSecretKey":"mock-aws-secret-key","sessionToken":"mock-aws-session-token"},"outputArtifacts":{"outputArtifacts":{"packaging":"ZIP","type":"LOCAL"}},"secondarySources":[],"sourceLocation":{"location":"","sourceIdentifier":"","type":"LOCAL"}}
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: localhost:8100
・・・
{"buildArn":"","metrics":[{"dimensions":[{"instance":"PROVISIONING","metricClass":"PhaseType"}],"metricValue":0,"name":"PhaseFailed","type":"COUNT"},{"dimensions":[{"instance":"PROVISIONING","metricClass":"PhaseType"}],"metricValue":0,"name":"PhaseFault","type":"COUNT"},{"dimensions":[{"instance":"PROVISIONING","metricClass":"PhaseType"}],"metricValue":1,"name":"PhaseSucceeded","type":"COUNT"},{"metricValue":0,"name":"EFS","type":"COUNT"},{"dimensions":[{"instance":"DOWNLOAD_SOURCE","metricClass":"PhaseType"}],"metricValue":0,"name":"PhaseCommandsNumber","type":"COUNT"},{"dimensions":[{"instance":"DOWNLOAD_SOURCE","metricClass":"PhaseType"}],"metricValue":0,"name":"PhaseFailed","type":"COUNT"},{"dimensions":[{"instance":"DOWNLOAD_SOURCE","metricClass":"PhaseType"}],"metricValue":0,"name":"PhaseFault","type":"COUNT"},{"dimensions":[{"instance":"DOWNLOAD_SOURCE","metricClass":"PhaseType"}],"metricValue":1,"name":"PhaseSucceeded","type":"COUNT"},{"dimensions":[{"instance":"INSTALL","metricClass":"PhaseType"}],"metricValue":0,"name":"PhaseCommandsNumber","type":"COUNT"},{"dimensions":[{"instance":"INSTALL","metricClass":"PhaseType"}],"metricValue":0,"name":"PhaseFailed","type":"COUNT"},{"dimensions":[{"instance":"INSTALL","metricClass":"PhaseType"}],"metricValue":0,"name":"PhaseFault","type":"COUNT"},{"dimensions":[{"instance":"INSTALL","metricClass":"PhaseType"}],"metricValue":1,"name":"PhaseSucceeded","type":"COUNT"},{"dimensions":[{"instance":"PRE_BUILD","metricClass":"PhaseType"}],"metricValue":0,"name":"PhaseCommandsNumber","type":"COUNT"},{"dimensions":[{"instance":"PRE_BUILD","metricClass":"PhaseType"}],"metricValue":0,"name":"PhaseFailed","type":"COUNT"},{"dimensions":[{"instance":"PRE_BUILD","metricClass":"PhaseType"}],"metricValue":0,"name":"PhaseFault","type":"COUNT"},{"dimensions":[{"instance":"PRE_BUILD","metricClass":"PhaseType"}],"metricValue":1,"name":"PhaseSucceeded","type":"COUNT"},{"dimensions":[{"instance":"BUILD","metricClass":"PhaseType"}],"metricValue":1,"name":"PhaseCommandsNumber","type":"COUNT"},{"dimensions":[{"instance":"BUILD","metricClass":"PhaseType"}],"metricValue":0,"name":"PhaseFailed","type":"COUNT"},{"dimensions":[{"instance":"BUILD","metricClass":"PhaseType"}],"metricValue":0,"name":"PhaseFault","type":"COUNT"},{"dimensions":[{"instance":"BUILD","metricClass":"PhaseType"}],"metricValue":1,"name":"PhaseSucceeded","type":"COUNT"},{"dimensions":[{"instance":"POST_BUILD","metricClass":"PhaseType"}],"metricValue":0,"name":"PhaseCommandsNumber","type":"COUNT"},{"dimensions":[{"instance":"POST_BUILD","metricClass":"PhaseType"}],"metricValue":0,"name":"PhaseFailed","type":"COUNT"},{"dimensions":[{"instance":"POST_BUILD","metricClass":"PhaseType"}],"metricValue":0,"name":"PhaseFault","type":"COUNT"},{"dimensions":[{"instance":"POST_BUILD","metricClass":"PhaseType"}],"metricValue":1,"name":"PhaseSucceeded","type":"COUNT"},{"dimensions":[{"instance":"UPLOAD_ARTIFACTS","metricClass":"PhaseType"}],"metricValue":0,"name":"PhaseFailed","type":"COUNT"},{"dimensions":[{"instance":"UPLOAD_ARTIFACTS","metricClass":"PhaseType"}],"metricValue":0,"name":"PhaseFault","type":"COUNT"},{"dimensions":[{"instance":"UPLOAD_ARTIFACTS","metricClass":"PhaseType"}],"metricValue":1,"name":"PhaseSucceeded","type":"COUNT"},{"dimensions":[{"instance":"go-git","metricClass":"GitLibrary"}],"metricValue":0,"name":"SucceededFailed","type":"COUNT"},{"dimensions":[{"instance":"go-git","metricClass":"GitLibrary"}],"metricValue":1,"name":"SucceededSucceeded","type":"COUNT"}]}
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
・・・
-----------------------------------------------------
****/**/** **:**:** {}
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: localhost:8100
・・・
{"agentFeatures":["ON_FAILURE_KEY_BUILDSPEC"],"buildArn":""}
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
・・・
-----------------------------------------------------
****/**/** **:**:** {"agentCredentials":{"awsAccessKey":"mock-aws-access-key","awsSecretKey":"mock-aws-secret-key","sessionToken":"mock-aws-session-token"},"outputArtifacts":{"outputArtifacts":{"packaging":"ZIP","type":"LOCAL"}},"secondarySources":[],"sourceLocation":{"location":"","sourceIdentifier":"","type":"LOCAL"}}
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: localhost:8100
・・・
{"buildArn":"","completedPhaseState":{"contexts":[{"containerType":"build_agent","message":"","statusCode":""}],"outputs":[],"phase":"FINALIZING","state":"SUCCEEDED"}}
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
・・・
-----------------------------------------------------
****/**/** **:**:** {"nextPhase":"COMPLETED"}

ただ、build側のentrypointで実行する/codebuild/readonly/bin/executorは一部デバッグ情報は入っているようなものの、gdbをかけても結局どんなことをしているかわからなかった。
おそらくagentコンテナを立ち上げなくても、ビルドコンテナ内で上記のようなレスポンスを返すサーバーを建ててしまえばホスト名の修正をしなくてもビルドは進む気がするが、わざわざそこまでやる意義はあまり感じられないので現状は対処方法の通りで十分である。

またpublic.ecr.aws/codebuild/local-buildsのメンテナンスはタグの最終更新が1年前だったり、ドキュメントの分量も他に比べると少なかったりで内部でもあまり頻繁にはされていないようにみえる。
その影響のためか、今回のように根本的にはおそらくホストのDocker環境が原因の問題が顕在化したと思われる。

TL;DR

背景

対処方法

docker-compose.ymlのバージョン

Waiting for agent pingから進まない

ビルド側コンテナでの名前解決失敗

修正後の実行結果

Discussion