Closed10

GitHub Actionsでsam deployしたLambdaが動かない問題を解決する

miztchmiztch
  • GitHub ActionsでSAMアプリケーションをデプロイする
  • LambdaのランタイムはGo(provided.al2)

デプロイは正常に終了したが、Lambdaを実行したら Runtime.ExitError

INIT_START Runtime Version: provided:al2.v28    Runtime Version ARN: arn:aws:lambda:ap-northeast-1::runtime:8bdc1e5642bbcd08528c3948a723b8901ec98ab986e6bab636fae24882de1d6c
/var/task/bootstrap: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by /var/task/bootstrap)
/var/task/bootstrap: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by /var/task/bootstrap)
INIT_REPORT Init Duration: 8.38 ms    Phase: init    Status: error    Error Type: Runtime.ExitError
/var/task/bootstrap: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by /var/task/bootstrap)
/var/task/bootstrap: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by /var/task/bootstrap)
INIT_REPORT Init Duration: 91.54 ms    Phase: invoke    Status: error    Error Type: Runtime.ExitError
START RequestId: 14aca2ee-d2d5-4b42-b89e-11521998d45f Version: $LATEST
RequestId: 14aca2ee-d2d5-4b42-b89e-11521998d45f Error: Runtime exited with error: exit status 1
Runtime.ExitError
END RequestId: 14aca2ee-d2d5-4b42-b89e-11521998d45f
REPORT RequestId: 14aca2ee-d2d5-4b42-b89e-11521998d45f    Duration: 112.10 ms    Billed Duration: 113 ms    Memory Size: 128 MB    Max Memory Used: 3 MB 
miztchmiztch

ローカル(WSL, ubuntu20.04)でビルド・デプロイする時は問題ないので、環境差分(というかActionsのRunnerの問題)だと推測

ビルドする時に --use-container付ければ良いのでは?と思って付けてみる

→ ビルドが失敗

aws_lambda_builders.exceptions.WorkflowFailedError: GoModulesBuilder:Resolver - Path resolution for runtime: provided of binary: go was not successful
GitHub Actionのログ(debug含んで長いので折り畳み)
  sam build --use-container
  shell: /usr/bin/bash -e {0}
  env:
    AWS_DEFAULT_REGION: ***
    AWS_REGION: ***
    AWS_ACCESS_KEY_ID: ***
    AWS_SECRET_ACCESS_KEY: ***
    AWS_SESSION_TOKEN: ***

	SAM CLI now collects telemetry to better understand customer needs.

	You can OPT OUT and disable telemetry collection by setting the
	environment variable SAM_CLI_TELEMETRY=0 in your shell.
	Thanks for your help!

	Learn More: https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-telemetry.html

2023-12-21 17:10:41,363 | Using config file: samconfig.toml, config environment: default
2023-12-21 17:10:41,363 | Expand command line arguments to:
2023-12-21 17:10:41,363 | --template_file=/home/runner/work/***Api/***Api/template.yaml --use_container --mount_with=READ --build_dir=.aws-sam/build --cache_dir=.aws-sam/cache 
2023-12-21 17:10:41,399 | 'build' command is called
2023-12-21 17:10:41,399 | Starting Build inside a container
2023-12-21 17:10:41,409 | Collected default values for parameters: {}
2023-12-21 17:10:41,434 | There is no customer defined id or cdk path defined for resource RestApi, so we will use the resource logical id as the resource id
2023-12-21 17:10:41,434 | There is no customer defined id or cdk path defined for resource Function, so we will use the resource logical id as the resource id
2023-12-21 17:10:41,434 | There is no customer defined id or cdk path defined for resource FunctionLogGroup, so we will use the resource logical id as the resource id
2023-12-21 17:10:41,482 | Unable to resolve property Value: {'Fn::GetAtt': ['Function', 'Arn']}. Leaving as is.
2023-12-21 17:10:41,483 | 0 stacks found in the template
2023-12-21 17:10:41,483 | Collected default values for parameters: {}
2023-12-21 17:10:41,503 | There is no customer defined id or cdk path defined for resource RestApi, so we will use the resource logical id as the resource id
2023-12-21 17:10:41,503 | There is no customer defined id or cdk path defined for resource Function, so we will use the resource logical id as the resource id
2023-12-21 17:10:41,503 | There is no customer defined id or cdk path defined for resource FunctionLogGroup, so we will use the resource logical id as the resource id
2023-12-21 17:10:41,549 | Unable to resolve property Value: {'Fn::GetAtt': ['Function', 'Arn']}. Leaving as is.
2023-12-21 17:10:41,549 | 3 resources found in the stack 
2023-12-21 17:10:41,549 | Found Serverless function with name='Function' and CodeUri='cmd/lambda/'
2023-12-21 17:10:41,549 | --base-dir is not presented, adjusting uri cmd/lambda/ relative to /home/runner/work/***Api/***Api/template.yaml
2023-12-21 17:10:41,560 | 3 resources found in the stack 
2023-12-21 17:10:41,560 | Found Serverless function with name='Function' and CodeUri='cmd/lambda/'
2023-12-21 17:10:41,561 | Instantiating build definitions
2023-12-21 17:10:41,561 | No previous build graph found, generating new one
2023-12-21 17:10:41,561 | Unique function build definition found, adding as new (Function Build Definition: BuildDefinition(provided.al2, /home/runner/work/***Api/***Api/cmd/lambda, Zip, , 12e086fe-aff4-4b7d-8336-09422fb0a8ba, {'BuildMethod': 'go1.x'}, {}, x86_64, []), Function: Function(function_id='Function', name='Function', functionname='Env-***-api', runtime='provided.al2', memory=128, timeout=30, handler='bootstrap', imageuri=None, packagetype='Zip', imageconfig=None, codeuri='/home/runner/work/***Api/***Api/cmd/lambda', environment={'Variables': {'***_TABLE_NAME': 'Env-***', 'COIN_API_URL': 'CoinApiUrl'}}, rolearn=None, layers=[], events={'Post***s': {'Type': 'Api', 'Properties': {'Path': '/v1/***s', 'Method': 'post', 'RestApiId': 'RestApi'}}, 'Get***s': {'Type': 'Api', 'Properties': {'Path': '/v1/***s/{proxy+}', 'Method': 'get', 'RestApiId': 'RestApi'}}, 'Patch***': {'Type': 'Api', 'Properties': {'Path': '/v1/***s/{proxy+}', 'Method': 'patch', 'RestApiId': 'RestApi'}}, 'Delete***': {'Type': 'Api', 'Properties': {'Path': '/v1/***s/{proxy+}', 'Method': 'delete', 'RestApiId': 'RestApi'}}, 'Post***Tickets': {'Type': 'Api', 'Properties': {'Path': '/v1/***s/{proxy+}/tickets', 'Method': 'post', 'RestApiId': 'RestApi'}}, 'Post***Results': {'Type': 'Api', 'Properties': {'Path': '/v1/***s/{proxy+}/results', 'Method': 'post', 'RestApiId': 'RestApi'}}, 'Get***Odds': {'Type': 'Api', 'Properties': {'Path': '/v1/***s/{proxy+}/odds/{proxy+}', 'Method': 'get', 'RestApiId': 'RestApi'}}}, metadata={'BuildMethod': 'go1.x', 'SamResourceId': 'Function'}, inlinecode=None, codesign_config_arn=None, architectures=['x86_64'], function_url_config=None, function_build_info=<FunctionBuildInfo.BuildableZip: ('BuildableZip', 'Regular ZIP function which can be build with SAM CLI')>, stack_path='', runtime_management_config=None, logging_config=None))
2023-12-21 17:10:41,563 | Building codeuri: /home/runner/work/***Api/***Api/cmd/lambda runtime: provided.al2 metadata: {'BuildMethod': 'go1.x'} architecture: x86_64 functions: Function
2023-12-21 17:10:41,563 | Building to following folder /home/runner/work/***Api/***Api/.aws-sam/build/Function
2023-12-21 17:10:41,573 | Checking free port on 127.0.0.1:5723

Fetching public.ecr.aws/sam/build-provided.al2:latest-x86_64 Docker container image...................................................................................................................................................................................................................................................................................................................................
2023-12-21 17:11:17,889 | Mounting /home/runner/work/***Api/***Api/cmd/lambda as /tmp/samcli/source:ro,delegated, inside runtime container
Using the request object from command line argument
Loading workflow module 'aws_lambda_builders.workflows'
Registering workflow 'CustomMakeBuilder' with capability 'Capability(language='provided', dependency_manager=None, application_framework=None)'
Registering workflow 'DotnetCliPackageBuilder' with capability 'Capability(language='dotnet', dependency_manager='cli-package', application_framework=None)'
Registering workflow 'GoModulesBuilder' with capability 'Capability(language='go', dependency_manager='modules', application_framework=None)'
Registering workflow 'JavaGradleWorkflow' with capability 'Capability(language='java', dependency_manager='gradle', application_framework=None)'
Registering workflow 'JavaMavenWorkflow' with capability 'Capability(language='java', dependency_manager='maven', application_framework=None)'
Registering workflow 'NodejsNpmBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm', application_framework=None)'
Registering workflow 'NodejsNpmEsbuildBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm-esbuild', application_framework=None)'
Registering workflow 'PythonPipBuilder' with capability 'Capability(language='python', dependency_manager='pip', application_framework=None)'
Registering workflow 'RubyBundlerBuilder' with capability 'Capability(language='ruby', dependency_manager='bundler', application_framework=None)'
Registering workflow 'RustCargoLambdaBuilder' with capability 'Capability(language='rust', dependency_manager='cargo', application_framework=None)'
Found workflow 'GoModulesBuilder' to support capabilities 'Capability(language='go', dependency_manager='modules', application_framework=None)'
Workflow GoModulesBuilder does not support value "False" for building in source. Using default value "True".
Builder workflow failed
Traceback (most recent call last):
  File "/usr/local/opt/sam-cli/lib64/python3.8/site-packages/aws_lambda_builders/workflow.py", line 77, in wrapper
    binary_checker.resolver.exec_paths
  File "/usr/local/opt/sam-cli/lib64/python3.8/site-packages/aws_lambda_builders/path_resolver.py", line 33, in exec_paths
    return self._which()
  File "/usr/local/opt/sam-cli/lib64/python3.8/site-packages/aws_lambda_builders/path_resolver.py", line 26, in _which
    raise ValueError(
ValueError: Path resolution for runtime: provided of binary: go was not successful

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/opt/sam-cli/lib64/python3.8/site-packages/aws_lambda_builders/__main__.py", line 117, in main
    builder.build(
  File "/usr/local/opt/sam-cli/lib64/python3.8/site-packages/aws_lambda_builders/builder.py", line 170, in build
    return workflow.run()
  File "/usr/local/opt/sam-cli/lib64/python3.8/site-packages/aws_lambda_builders/workflow.py", line 82, in wrapper
    raise WorkflowFailedError(workflow_name=self.NAME, action_name="Resolver", reason=str(ex))
aws_lambda_builders.exceptions.WorkflowFailedError: GoModulesBuilder:Resolver - Path resolution for runtime: provided of binary: go was not successful
2023-12-21 17:11:19,082 | Build inside container returned response {"jsonrpc": "2.0", "id": 1, "error": {"code": 400, "message": "GoModulesBuilder:Resolver - Path resolution for runtime: provided of binary: go was not successful"}}
2023-12-21 17:11:19,230 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics

Build Failed
2023-12-21 17:11:19,238 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2023-12-21 17:11:19,238 | Unable to find Click Context for getting session_id.
2023-12-21 17:11:19,238 | Sending Telemetry: {'metrics': [{'events': {'requestId': 'c751d61a-364f-4465-9dbf-6538dac3379e', 'installationId': '8601e8e3-ee62-46a6-9587-a89c584c916f', 'sessionId': '5dcf3c68-54bf-4208-b669-d154082f349a', 'executionEnvironment': 'GitHubAction', 'ci': True, 'pyversion': '3.10.12', 'samcliVersion': '1.105.0', 'commandName': 'sam build', 'metricSpecificAttributes': {'events': [{'event_name': 'SamConfigFileExtension', 'event_value': '.toml', 'thread_id': '5c156de52fe94def8d18e8def1ae7267', 'time_stamp': '2023-12-21 17:10:41.327', 'exception_name': None}, {'event_name': 'SamConfigFileExtension', 'event_value': '.toml', 'thread_id': 'b2f6ebcf5af8499d8db5cc851691e95b', 'time_stamp': '2023-12-21 17:10:41.363', 'exception_name': None}, {'event_name': 'BuildWorkflowUsed', 'event_value': 'go-modules', 'thread_id': '43d531dc96534b47884ef5e767cdda56', 'time_stamp': '2023-12-21 17:10:41.560', 'exception_name': None}, {'event_name': 'BuildFunctionRuntime', 'event_value': 'provided.al2', 'thread_id': 'da98c071c50d412685101efa035dde55', 'time_stamp': '2023-12-21 17:10:41.561', 'exception_name': None}, {'event_name': 'BuildWorkflowUsed', 'event_value': 'go-modules', 'thread_id': '7b30bfc15c844bb2a307f427d8491381', 'time_stamp': '2023-12-21 17:10:41.563', 'exception_name': None}]}}}]}
2023-12-21 17:11:19,239 | Sending Telemetry: {'metrics': [{'commandRun': {'requestId': 'b582a011-8e46-4dd9-b375-bbc8d45e9805', 'installationId': '8601e8e3-ee62-46a6-9587-a89c584c916f', 'sessionId': '5dcf3c68-54bf-4208-b669-d154082f349a', 'executionEnvironment': 'GitHubAction', 'ci': True, 'pyversion': '3.10.12', 'samcliVersion': '1.105.0', 'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam build', 'metricSpecificAttributes': {'projectType': 'CFN', 'gitOrigin': None, 'projectName': '27caac2342bb08ffd25c9e2e1983a86f99dbf85d806ec2dc63e02d8e560d9fa6', 'initialCommit': None}, 'duration': 37866, 'exitReason': 'BuildInsideContainerError', 'exitCode': 1}}]}
2023-12-21 17:11:19,785 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)
Error: GoModulesBuilder:Resolver - Path resolution for runtime: provided of binary: go was not successful
2023-12-21 17:11:19,788 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)
Error: Process completed with exit code 1.
miztchmiztch

https://github.com/aws/aws-sam-build-images/pull/71

このイメージにGo入れろやのPRもあったが蹴られている
特定の言語を考慮したものはカスタムランタイムのイメージに入れません、というスタンスは理解はできる

This doesn't look right to me. We don't install any other runtimes, so I'm not sure we should do this for Go.

I think there might be a confusion on building Go with the provided runtime, but I'll get back to you after chatting with the team.

miztchmiztch

さっきのissueで専用のオレオレビルドイメージ作るやで!って書いてあったので、真似してみた

          sam build \
            --use-container \
            --build-image ghcr.io/miztch/sam-build-custom-image:v0.0.1-go1.21.5

ビルドしようとしたので前進はしたが、プロジェクトのディレクトリ構成の問題で蹴られる
(ただローカルでビルドすれば通るので……でわけわからなくなった)

Fetching ghcr.io/miztch/sam-build-custom-image:v0.0.1-go1.21.5 Docker container image.....................................................................................................................................................................................................................................................................................................................................................................................
2023-12-22 17:21:21,594 | Mounting /home/runner/work/***Api/***Api/cmd/lambda as /tmp/samcli/source:ro,delegated, inside runtime container
Using the request object from command line argument
Loading workflow module 'aws_lambda_builders.workflows'
Registering workflow 'CustomMakeBuilder' with capability 'Capability(language='provided', dependency_manager=None, application_framework=None)'
Registering workflow 'DotnetCliPackageBuilder' with capability 'Capability(language='dotnet', dependency_manager='cli-package', application_framework=None)'
Registering workflow 'GoModulesBuilder' with capability 'Capability(language='go', dependency_manager='modules', application_framework=None)'
Registering workflow 'JavaGradleWorkflow' with capability 'Capability(language='java', dependency_manager='gradle', application_framework=None)'
Registering workflow 'JavaMavenWorkflow' with capability 'Capability(language='java', dependency_manager='maven', application_framework=None)'
Registering workflow 'NodejsNpmBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm', application_framework=None)'
Registering workflow 'NodejsNpmEsbuildBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm-esbuild', application_framework=None)'
Registering workflow 'PythonPipBuilder' with capability 'Capability(language='python', dependency_manager='pip', application_framework=None)'
Registering workflow 'RubyBundlerBuilder' with capability 'Capability(language='ruby', dependency_manager='bundler', application_framework=None)'
Registering workflow 'RustCargoLambdaBuilder' with capability 'Capability(language='rust', dependency_manager='cargo', application_framework=None)'
Found workflow 'GoModulesBuilder' to support capabilities 'Capability(language='go', dependency_manager='modules', application_framework=None)'
Workflow GoModulesBuilder does not support value "False" for building in source. Using default value "True".
Running workflow 'GoModulesBuilder'
 Running GoModulesBuilder:Build
go: go.mod file not found in current directory or any parent directory; see 'go help modules'
Go files not found. Attempting to build for Go files in a different directory
Go files not found at CodeUri /tmp/samcli/source . Descending into sub-directories to find the handler: /tmp/samcli/source/bootstrap
GoModulesBuilder:Build failed
Traceback (most recent call last):
  File "/usr/local/opt/sam-cli/lib64/python3.8/site-packages/aws_lambda_builders/workflows/go_modules/actions.py", line 22, in execute
    self.builder.build(self.source_dir, self.output_path)
  File "/usr/local/opt/sam-cli/lib64/python3.8/site-packages/aws_lambda_builders/workflows/go_modules/builder.py", line 77, in build
    raise BuilderError(message=p_err.decode("utf8").strip())
aws_lambda_builders.workflows.go_modules.builder.BuilderError: Builder Failed: go: go.mod file not found in current directory or any parent directory; see 'go help modules'
miztchmiztch

ここで当初のエラーに立ち返る

https://ebc-2in2crc.hatenablog.jp/entry/2023/01/17/192508
https://ebc-2in2crc.hatenablog.jp/entry/2020/10/14/030150

エラーメッセージから上記がヒット

go build するマシンの glibc のバージョンが go build した Go バイナリを動かすマシンの glibc のバージョンより新しいとこの事象が発生するようなので go build するマシンの glibc のバージョンを下げるか go build した Go バイナリを動かすマシンの glibc のバージョンを上げるかしたら事象は解消しそう。

glibcのバージョン調べるのめんどくないか?俺はLambda動かしたいだけだが

miztchmiztch

ひとまず調べた

GitHub Actions

  • 2023/12現在選べるLinuxのRunnerは、ubuntu-20.04 or ubuntu-22.04

https://docs.github.com/ja/actions/using-github-hosted-runners/about-github-hosted-runners/about-github-hosted-runners#supported-runners-and-hardware-resources

https://zenn.dev/msk/scraps/98ab4b21d2c0d1

上記から引用

OS バージョン glibcのバージョン
Ubuntu 20.04 2.31
Ubuntu 22.04 2.35

Amazon Linux 2

https://aws.amazon.com/jp/amazon-linux-2/release-notes/

リリースノートで確認できるアップデートを見るとglibcは2.26らしい
(Amazon Linux 2.0.20211103.0)

glibc-2.26-56.amzn2.aarch64
glibc-2.26-56.amzn2.x86_64

Amazon Linux 2023

glibcは2.34
https://docs.aws.amazon.com/ja_jp/linux/al2023/ug/core-glibc.html

miztchmiztch

GitHub Actionsのworkflowでは ubuntu-latest を利用していたので、glibcのバージョンが新しすぎたということになる…

ubuntu-20.04provided.al2023 の組み合わせであればこれをクリアできそうなので、

  • sam buildをランナー上で実行
  • ランナーのイメージはubuntu-20.04
  • Lambdaのランタイムはprovided.al2023

でうまくいく気がする

miztchmiztch

ubuntu-20.04 と provided.al2023 の組み合わせ

でデプロイ成功し、Lambdaも動いた

miztchmiztch

ランタイムが provided.al2 でもなぜか動いている……?一旦解決したのでヨシ

このスクラップは4ヶ月前にクローズされました