RustをAWS Lambda + API Gateway環境で動かせるようにするまでのメモ

Ash

Rust初心者がSAMを使ってAWSにAPIを公開するまでやってみるメモ
環境メモ

M1 Max Mac Book Pro 2021
macOS Monterey 12.3.1
VSCode 1.66

Ash

まずは、RustのRemote-Containers環境を作る。
（コンテナサイズは考慮してない）

Remote-Containersのコマンド

この時点でのdevcontainer.json

devcontainer.json

// For format details, see https://aka.ms/devcontainer.json. For config options, see the README at:
// https://github.com/microsoft/vscode-dev-containers/tree/v0.231.6/containers/ubuntu
{
	"name": "Ubuntu",
	"build": {
		"dockerfile": "Dockerfile",
		// Update 'VARIANT' to pick an Ubuntu version: hirsute, focal, bionic
		// Use hirsute or bionic on local arm64/Apple Silicon.
		"args": { "VARIANT": "bionic" }
	},

	// Set *default* container specific settings.json values on container create.
	"settings": {},


	// Add the IDs of extensions you want installed when the container is created.
	"extensions": [],

	// Use 'forwardPorts' to make a list of ports inside the container available locally.
	// "forwardPorts": [],

	// Use 'postCreateCommand' to run commands after the container is created.
	// "postCreateCommand": "uname -a",

	// Comment out to connect as root instead. More info: https://aka.ms/vscode-remote/containers/non-root.
	"remoteUser": "vscode",
	"features": {
		"git": "latest",
		"aws-cli": "latest",
		"rust": "latest"
	}
}

Ash

とりあえずRustを動かしてみる。

$ cargo new rust-lambda
     Created binary (application) `rust-lambda` package
$ cd rust-lambda
$ cargo run
   Compiling rust-lambda v0.1.0 (/workspaces/rust-aws-sam/rust-lambda)
    Finished dev [unoptimized + debuginfo] target(s) in 3.93s
     Running `target/debug/rust-lambda`
Hello, world!

Ash

Lambda単独で動かせるようにする。

Cargo.toml

[package]
name = "rust-lambda"
version = "0.1.0"
edition = "2021"
license = "MIT"
autobins = false

[[bin]]
name = "bootstrap"
path = "src/main.rs"

[dependencies]
lambda_runtime = "*"
serde = "*"
serde_json = "*"
serde_derive = "*"
log = "*"
simple_logger = "*"

main.rs

#[macro_use]
extern crate lambda_runtime as lambda;
#[macro_use]
extern crate serde_derive;
#[macro_use]
extern crate log;
extern crate simple_logger;

use lambda::error::HandlerError;

use std::error::Error;

#[derive(Deserialize, Clone)]
struct CustomEvent {
    #[serde(rename = "firstName")]
    first_name: String,
}

#[derive(Serialize, Clone)]
struct CustomOutput {
    message: String,
}

fn main() -> Result<(), Box<dyn Error>> {
    simple_logger::init_with_level(log::Level::Info)?;
    lambda!(my_handler);

    Ok(())
}

fn my_handler(e: CustomEvent, c: lambda::Context) -> Result<CustomOutput, HandlerError> {
    if e.first_name == "" {
        error!("Empty first name in request {}", c.aws_request_id);
        return Err(c.new_error("Empty first name"));
    }

    Ok(CustomOutput {
        message: format!("Hello, {}!", e.first_name),
    })
}

クロスコンパイル用の設定

$ rustup target add x86_64-unknown-linux-musl

$ brew install filosottile/musl-cross/musl-cross

トラブル発生 -> arm64向けのhomebrewがインストールできない

Ash

上記のコマンドはMac向けだったので、Ubuntuでは別の方法で回避してみる。

cargo install cross

cross build --target x86_64-unknown-linux-musl

めっちゃエラーがでた。

さらに別の方法

main.rs

// This example requires the following input to succeed:
// { "command": "do something" }

use lambda_runtime::{service_fn, Error, LambdaEvent};
use serde::{Deserialize, Serialize};

/// This is also a made-up example. Requests come into the runtime as unicode
/// strings in json format, which can map to any structure that implements `serde::Deserialize`
/// The runtime pays no attention to the contents of the request payload.
#[derive(Deserialize)]
struct Request {
    command: String,
}

/// This is a made-up example of what a response structure may look like.
/// There is no restriction on what it can be. The runtime requires responses
/// to be serialized into json. The runtime pays no attention
/// to the contents of the response payload.
#[derive(Serialize)]
struct Response {
    req_id: String,
    msg: String,
}

#[tokio::main]
async fn main() -> Result<(), Error> {
    // tracing_subscriber::fmt()
    //     .with_max_level(tracing::Level::INFO)
    //     // this needs to be set to false, otherwise ANSI color codes will
    //     // show up in a confusing manner in CloudWatch logs.
    //     .with_ansi(false)
    //     // disabling time is handy because CloudWatch will add the ingestion time.
    //     .without_time()
    //     .init();

    let func = service_fn(my_handler);
    lambda_runtime::run(func).await?;
    Ok(())
}

pub(crate) async fn my_handler(event: LambdaEvent<Request>) -> Result<Response, Error> {
    // extract some useful info from the request
    let command = event.payload.command;

    // prepare the response
    let resp = Response {
        req_id: event.context.request_id,
        msg: format!("Command {} executed.", command),
    };

    // return `Response` (it will be serialized to JSON automatically by the runtime)
    Ok(resp)
}

Cargo.toml

[package]
name = "rust-lambda"
version = "0.1.0"
edition = "2021"
license = "MIT"
autobins = false

[[bin]]
name = "bootstrap"
path = "src/main.rs"

[dependencies]
tokio = { version = "*", features = ["macros", "io-util", "sync", "rt-multi-thread"] }
tracing = { version = "*", features = ["log"] }
lambda_runtime = "*"
serde = "*"
serde_json = "*"
serde_derive = "*"
log = "*"
simple_logger = "*"

[dev-dependencies]
tracing-subscriber = "*"
simple-error = "*"

$ cargo build --release --target x86_64-unknown-linux-musl 
   Compiling libc v0.2.123
   Compiling cfg-if v1.0.0
   Compiling proc-macro2 v1.0.37
   Compiling unicode-xid v0.2.2
   Compiling syn v1.0.91
error[E0463]: can't find crate for `core`
  |
  = note: the `x86_64-unknown-linux-musl` target may not be installed
  = help: consider downloading the target with `rustup target add x86_64-unknown-linux-musl`

error[E0463]: can't find crate for `compiler_builtins`

For more information about this error, try `rustc --explain E0463`.
error: could not compile `cfg-if` due to 2 previous errors
warning: build failed, waiting for other jobs to finish...
error: build failed

うむ…

rustup target add x86_64-unknown-linux-musl
info: downloading component 'rust-std' for 'x86_64-unknown-linux-musl'
info: installing component 'rust-std' for 'x86_64-unknown-linux-musl'
 39.7 MiB /  39.7 MiB (100 %)  18.9 MiB/s in  2s ETA:  0s

ライブラリのコンパイルは上手く行ったが、自分のソースコードのコンパイルでエラーがでた。

   Compiling rust-lambda v0.1.0 (/workspaces/rust-aws-sam/rust-lambda)
WARN rustc_codegen_ssa::back::link Linker does not support -static-pie command line option. Retrying with -static instead.
error: linking with `cc` failed: exit status: 1

$ mkdir .cargo
$ echo '[target.x86_64-unknown-linux-musl]
> linker = "x86_64-linux-musl-gcc"' > .cargo/config

$ export RUSTFLAGS='-C linker=x86_64-linux-gnu-gcc'
$ sudo apt-get update -y
$ sudo apt-get install -y gcc-x86-64-linux-gnu

上記でとりあえずコンパイルはできた。
RUSTFLAGSを設定しなくても、.cargo/configの設定でもいける。

linker = "x86_64-linux-gnu-gcc"

Ash

いったんビルドまでのまとめ

$ rustup target add x86_64-unknown-linux-musl
$ sudo apt-get update -y
$ sudo apt-get install -y gcc-x86-64-linux-gnu

$ mkdir .cargo
$ echo '[target.x86_64-unknown-linux-musl]
> linker = "x86_64-linux-gnu-gcc"' > .cargo/config

cargo build --release --target x86_64-unknown-linux-musl

Ash

デプロイ用にzipを作る

$ zip -j rust.zip ./target/x86_64-unknown-linux-musl/release/bootstrap

とりあえず、手抜きでコンソールからzipをアップロードして動作確認したら動いた！

Request

{ "command": "do something" }

Response

{
  "req_id": "492366fe-7ae4-4fbc-8d79-527872b87496",
  "msg": "Command do something executed."
}

Ash

ってか、久しぶりにLambdaを見たらarm64にも対応してるんじゃね？

x86_64より安価で高速ならarm64を選ぶよね？

Ash

arm64を試す。

.cargo/config.toml

[target.aarch64-unknown-linux-gnu]
rustflags = [
  "-C", "target-cpu=neoverse-n1",
]

[target.x86_64-unknown-linux-musl]
linker = "x86_64-linux-gnu-gcc"

cargo build --release --target aarch64-unknown-linux-gnu

zip -j rust_arm64.zip ./target/aarch64-unknown-linux-gnu/release/bootstrap

arm64でも普通に動いたw

Ash

awscliを使いやすくするために.devcontainerでdocker-composeを使うように変更。

devcontainer.json

// For format details, see https://aka.ms/devcontainer.json. For config options, see the README at:
// https://github.com/microsoft/vscode-dev-containers/tree/v0.231.6/containers/ubuntu
{
	"name": "Ubuntu",

	// Update the 'dockerComposeFile' list if you have more compose files or use different names.
	// The .devcontainer/docker-compose.yml file contains any overrides you need/want to make.
	"dockerComposeFile": [
		"docker-compose.yml"
	],

	// The 'service' property is the name of the service for the container that VS Code should
	// use. Update this value and .devcontainer/docker-compose.yml to the real service name.
	"service": "rust",

	// The optional 'workspaceFolder' property is the path VS Code should open by default when
	// connected. This is typically a file mount in .devcontainer/docker-compose.yml
	"workspaceFolder": "/workspace",

	// "build": {
	// 	"dockerfile": "Dockerfile",
	// 	// Update 'VARIANT' to pick an Ubuntu version: hirsute, focal, bionic
	// 	// Use hirsute or bionic on local arm64/Apple Silicon.
	// 	"args": { "VARIANT": "bionic" }
	// },

	// Set *default* container specific settings.json values on container create.
	"settings": {},


	// Add the IDs of extensions you want installed when the container is created.
	"extensions": [],

	// Use 'forwardPorts' to make a list of ports inside the container available locally.
	// "forwardPorts": [],

	// Use 'postCreateCommand' to run commands after the container is created.
	// "postCreateCommand": "uname -a",

	// Comment out to connect as root instead. More info: https://aka.ms/vscode-remote/containers/non-root.
	"remoteUser": "vscode",
	"features": {}
}

docker-compose.yml

version: "3"

services:
  rust:
    build: 
      context: ./
      dockerfile: Dockerfile
      args:
        - VARIANT=hirsute
    working_dir: /workspace
    volumes:
      - ~/.aws:/home/vscode/.aws
      - ~/.ssh:/home/vscode/.ssh
      - ../:/workspace:delegated
    # environment:
      # - HOGE=value

    # Overrides default command so things don't shut down after the process ends.
    command: /bin/sh -c "while sleep 1000; do :; done"

ホスト環境の.awsと.sshをそのまま使う。

Ash

Dockerfileでawcliやらrustの実行に必要なコンポーネントをインストールできるように変更。

Dockerfile

# See here for image contents: https://github.com/microsoft/vscode-dev-containers/tree/v0.231.6/containers/ubuntu/.devcontainer/base.Dockerfile

ARG VARIANT
FROM mcr.microsoft.com/vscode/devcontainers/base:0-${VARIANT}

RUN apt-get update \
    && apt-get -y install --no-install-recommends \
     gcc \ 
     build-essential \
     python3 python3-pip \
     git \
     gcc-x86-64-linux-gnu \
    && apt-get -y clean \
    && rm -rf /var/lib/apt/lists/*

# aws-cli v2
RUN curl "https://awscli.amazonaws.com/awscli-exe-linux-aarch64.zip" -o "awscliv2.zip"
RUN unzip awscliv2.zip
RUN ./aws/install

# aws-sam-cli
RUN pip3 install --upgrade aws-sam-cli

USER vscode

# Rust
RUN curl https://sh.rustup.rs -sSf | bash -s -- -y
ENV PATH $PATH:/home/vscode/.cargo/bin

RUN rustup target add x86_64-unknown-linux-musl

Rustのインストールは実行ユーザーをvscodeにしないと上手く動かなかった。
デフォルトだと/root配下に入るのでアクセスできない。
（他にもよい方法があるかもしれん）

Ash

samで色々ハマる。
arm64で動かしたかったので最初に設定したのはこれ。

  RustTestFunction:
    Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
    Properties:
      FunctionName: RustTestArm64
      Architectures: [ arm64 ] 
      Handler: main
      Runtime: provided
      CodeUri: .
      Events:
      〜省略〜
    Metadata:
      BuildMethod: makefile

これで動かすとsam deploy時に失敗する。
"Runtime provided does not support the following architectures [arm64].
arm64の場合、Runtimeを以下のように設定しないといけないらしい。

      Runtime: provided.al2

Ash

Makefileでもハマる。
Error: CustomMakeBuilder:MakeBuild - Make Failed: /workspace/Makefile:2: *** missing separator. Stop.
知ってる人なら常識なのだろうが、Makefileのコマンド部分はタブで区切らないとダメなそうな

build-RustTestFunction:
	cargo build --release --target aarch64-unknown-linux-gnu
	cp ./target/aarch64-unknown-linux-gnu/release/bootstrap $(ARTIFACTS_DIR)

Ash

うーん、bionicでbuildした奴なら動くけど、hirsuteだとダメっぽいな。

START RequestId: 5e65578e-5e14-4a98-a5d7-0746bd0f6c92 Version: $LATEST
/var/task/bootstrap: /lib64/libc.so.6: version `GLIBC_2.28' not found (required by /var/task/bootstrap)
/var/task/bootstrap: /lib64/libc.so.6: version `GLIBC_2.33' not found (required by /var/task/bootstrap)
/var/task/bootstrap: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by /var/task/bootstrap)
/var/task/bootstrap: /lib64/libc.so.6: version `GLIBC_2.28' not found (required by /var/task/bootstrap)
/var/task/bootstrap: /lib64/libc.so.6: version `GLIBC_2.33' not found (required by /var/task/bootstrap)
/var/task/bootstrap: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by /var/task/bootstrap)
END RequestId: 5e65578e-5e14-4a98-a5d7-0746bd0f6c92

色々と謎だな…

Ash

No space left on device に遭遇する。

$ df
Filesystem     1K-blocks      Used Available Use% Mounted on
overlay         61255492  58724724         0 100% /
tmpfs              65536         0     65536   0% /dev
shm                65536         0     65536   0% /dev/shm
/dev/vda1       61255492  58724724         0 100% /vscode
grpcfuse       971350180 536842192 434507988  56% /workspace
tmpfs            4070996         0   4070996   0% /sys/firmware

100%になったのはわかったが、対処方法がわからんのでDockerイメージを削除して再作成で対処。

Ash

bionicの場合、pythonが古くてsamのインストールでコケたのでpython3.9を入れる。

$ sudo apt update
$ sudo apt install build-essential libbz2-dev libdb-dev \
>   libreadline-dev libffi-dev libgdbm-dev liblzma-dev \
>   libncursesw5-dev libsqlite3-dev libssl-dev \
>   zlib1g-dev uuid-dev tk-dev

$ wget https://www.python.org/ftp/python/3.9.12/Python-3.9.12.tar.xz
$ tar xJf Python-3.9.12.tar.xz 
$ cd Python-3.9.12/
$ ./configure
$ make
$ sudo make install

Ash