💨
AWS入門 - Copilot 使ってみる２
AWSを使う必要が出てきたので、色々と試してみる。
ここでは、Copilot を試す。
AWS Copilot
前回、Copilotを軽く使ってみた。
色々と機能があるので、今回はApplication・Environment・Service周りについて、もう少し詳しく整理してみる。
Applications
copilot app initでApplicationだけを作成できる。
$ copilot app init
Application name: app-1120
✔ Created the infrastructure to manage services and jobs under application app-1120..
✔ The directory copilot will hold service manifests for application app-1120.
Recommended follow-up action:
Run `copilot init` to add a new service or job to your application.
$ copilot app ls
app-1120
$ cat copilot/.workspace
application: app-1120
ApplicationはAWSアカウントに紐付ける形で作成される。
これは、Systems ManagerのParameter Storeに保存されていて、Regionを跨いでApplicationはユニークな扱いとなっている。
また、Application内の各リソースには copilot-applicationタグが付与されていて、Applicationに紐付いているリソースであることが確認できたりする。
CloudFormationのStackが作成され、IAMロールが２つ作成されている。
まとめるとこんな感じ。
Environments
copilot env initでEnvironmentを作成できる。
使用するVPCや作成するCIDRを指定できるっぽいが、ここではデフォルト設定で進める。
$ copilot env init
Only found one application, defaulting to: app-1120
Environment name: dev
Credential source: [profile default]
Default environment configuration? Yes, use default.
✔ Linked account xxxxxxxxxxxx and region ap-northeast-1 to application app-1120..
✔ Proposing infrastructure changes for the app-1120-dev environment.
- Creating the infrastructure for the app-1120-dev environment. [create complete] [88.9s]
- An IAM Role for AWS CloudFormation to manage resources [create complete] [25.9s]
- An ECS cluster to group your services [create complete] [9.4s]
- Enable long ARN formats for the authenticated AWS principal [create complete] [2.6s]
- An IAM Role to describe resources in your environment [create complete] [26.9s]
- A security group to allow your containers to talk to each other [create complete] [3.9s]
- An Internet Gateway to connect to the public internet [create complete] [20.0s]
- Private subnet 1 for resources with no internet access [create complete] [16.6s]
- Private subnet 2 for resources with no internet access [create complete] [16.6s]
- Public subnet 1 for resources that can access the internet [create complete] [16.6s]
- Public subnet 2 for resources that can access the internet [create complete] [16.6s]
- A Virtual Private Cloud to control networking of your AWS resources [create complete] [16.8s]
✔ Created environment dev in region ap-northeast-1 under application app-1120.
$ copilot env init
Only found one application, defaulting to: app-1120
Environment name: prod
Credential source: [profile default]
Default environment configuration? Yes, use default.
✔ Linked account xxxxxxxxxxxx and region ap-northeast-1 to application app-1120..
✔ Proposing infrastructure changes for the app-1120-prod environment.
- Creating the infrastructure for the app-1120-prod environment. [create complete] [83.8s]
- An IAM Role for AWS CloudFormation to manage resources [create complete] [21.6s]
- An ECS cluster to group your services [create complete] [9.0s]
- Enable long ARN formats for the authenticated AWS principal [create complete] [2.0s]
- An IAM Role to describe resources in your environment [create complete] [26.3s]
- A security group to allow your containers to talk to each other [create complete] [5.8s]
- An Internet Gateway to connect to the public internet [create complete] [18.4s]
- Private subnet 1 for resources with no internet access [create complete] [15.8s]
- Private subnet 2 for resources with no internet access [create complete] [19.0s]
- Public subnet 1 for resources that can access the internet [create complete] [15.8s]
- Public subnet 2 for resources that can access the internet [create complete] [15.8s]
- A Virtual Private Cloud to control networking of your AWS resources [create complete] [15.8s]
✔ Created environment prod in region ap-northeast-1 under application app-1120.
$ copilot env ls
Only found one application, defaulting to: app-1120
prod
dev
Environementを作成すると前に、App Infrastructureと呼ばれる概念に必要なリソースも作成されている。
Application Infrastructure
Application内で共有して使うリソースが作祭されている。
ここでは、S3・KMS Keyが作成されていて、これらをApplication Infrastructureと呼ぶらしい。（後で作成されるECR Repositoryもこの概念に含まれる）
S3とKMS Keyは、Region・アカウントを跨いでCodePipelineからデプロイできるようにするものらしい。
Environment Infrastructure
App Infrastructureが作成された後に、Environmentが作成されている。
Application Infrastructureと同じく、Environment Infrastructureと呼ばれるEnvironment内で共有して使うリソースが作成される。
- IAM Role
- VPC・Subnet・Internet Gateway・Security Group ...
- ECS Cluster
- ...
なので、VPCやECS ClusterはEnvironment内の各Serviceで共有する形になる。
言い換えると、異なるEnvironmentであれば、VPCやECS Clusterは別になっていて、ネットワーク的にも分離されているということ。
また、Multi AZ（2AZ）となるようSubnetが作成され、各AZにはPublic SubnetとPrivate Subnetが１つずつ作成される。
Services
Service Type
Serviceには決められたタイプがあり、タイプに応じて構築されるシステムの構成が決まる。
Service Typeは今の所、次の４つである。
- Request-Driven Web Service (App Runner)
- Load Balanced Web Service (Internet to ECS on Fargate)
- Backend Service (ECS on Fargate)
- Worker Service (Events to SQS to ECS on Fargate)
ECSではなくApp Runnerが使われる場合はVPCを指定できないやつなので、Environment InfrastructureのVPCなどを共有して使うことができない。
Request-Driven Web Service
App Runnerをベースとして作成されるServiceタイプ。
インターネット公開して、HTTPでやりとりするような普通のWebサーバーとして使える。
また、オートスケールが有効になっていて、タイミングによってリクエスト数に差がある場合などはコスト的に優れる。
あと、App Runnerなので、CodePipelineなど面倒くさいやつを使わずに済む。
copilot svc initすることで、Serviceの定義ファイルと、それに対応したECR Repositoryが作成される。
その後にcopilot svc deployすることで、任意のEnvironmentへとServiceをデプロイできる。
次のリソースが作例される。
- ECR Repository
- IAM Role
- App Runner Service
$ copilot svc init
Note: It's best to run this command in the root of your workspace.
Service type: Request-Driven Web Service
Service name: svc-rdws
Dockerfile: ./Dockerfile
✔ Wrote the manifest for service svc-rdws at copilot/svc-rdws/manifest.yml
Your manifest contains configurations like your container size and port (:80).
✔ Created ECR repositories for service svc-rdws..
Recommended follow-up actions:
- Update your manifest copilot/svc-rdws/manifest.yml to change the defaults.
- Run `copilot svc deploy --name svc-rdws --env test` to deploy your service to a test environment.
$ copilot svc deploy --name svc-rdws --env dev
Environment dev is already on the latest version v1.6.1, skip upgrade.
Building your container image: docker build -t xxxxxxxxxxxx.dkr.ecr.ap-northeast-1.amazonaws.com/app-1120/svc-rdws --platform linux/amd64 /Users/xxxxxxxxxxxx/test-copilot -f /Users/xxxxxxxxxxxx/test-copilot/Dockerfile
[+] Building 3.0s (8/8) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 36B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/nginx:latest 2.9s
=> [auth] library/nginx:pull token for registry-1.docker.io 0.0s
=> [internal] load build context 0.0s
=> => transferring context: 61B 0.0s
=> [1/2] FROM docker.io/library/nginx:latest@sha256:097c3a0913d7e3a5b01b6c685a60c03632fc7a2b50bc8e35bcaa3691d788226e 0.0s
=> CACHED [2/2] COPY html /usr/share/nginx/html 0.0s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:d1d33bd912e15417ebfaef9f74bd7ac167e566c955ce3c3d529d36de49979ae4 0.0s
=> => naming to xxxxxxxxxxxx.dkr.ecr.ap-northeast-1.amazonaws.com/app-1120/svc-rdws 0.0s
Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
Login Succeeded
Logging in with your password grants your terminal complete access to your account.
For better security, log in with a limited-privilege personal access token. Learn more at https://docs.docker.com/go/access-tokens/
Using default tag: latest
The push refers to repository [xxxxxxxxxxxx.dkr.ecr.ap-northeast-1.amazonaws.com/app-1120/svc-rdws]
93ad08da2a86: Pushed
8525cde30b22: Pushed
1e8ad06c81b6: Pushed
49eeddd2150f: Pushed
ff4c72779430: Pushed
37380c5830fe: Pushed
e1bbcf243d0e: Pushed
latest: digest: sha256:8a8c0bf603c07d1443c73fa2cc396e8322dd14edc3ce464b9d42021a6c368d97 size: 1777
✔ Proposing infrastructure changes for stack app-1120-dev-svc-rdws
- Creating the infrastructure for stack app-1120-dev-svc-rdws [create complete] [292.9s]
- An IAM Role for App Runner to use on your behalf to pull your image from ECR [create complete] [23.2s]
- An IAM role to control permissions for the containers in your service [create complete] [26.7s]
- An App Runner service to run and manage your containers [create complete] [259.2s]
✔ Deployed service svc-rdws.
Recommended follow-up action:
You can access your service at https://qhgbmqm29u.ap-northeast-1.awsapprunner.com over the internet.
$ copilot svc deploy --name svc-rdws --env prod
...
✔ Proposing infrastructure changes for stack app-1120-prod-svc-rdws
- Creating the infrastructure for stack app-1120-prod-svc-rdws [create complete] [255.7s]
- An IAM Role for App Runner to use on your behalf to pull your image from ECR [create complete] [22.7s]
- An IAM role to control permissions for the containers in your service [create complete] [22.7s]
- An App Runner service to run and manage your containers [create complete] [222.0s]
✔ Deployed service svc-rdws.
Recommended follow-up action:
You can access your service at https://qakhamwrmv.ap-northeast-1.awsapprunner.com over the internet.
$ copilot svc ls
Name Type
---- ----
svc-rdws Request-Driven Web Service
$ copilot svc logs --env dev --name svc-rdws
instance/7418e88e7b7f4a40 2021/11/20 10:34:30 [notice] 1#1: nginx/1.21.4
instance/7418e88e7b7f4a40 2021/11/20 10:34:30 [notice] 1#1: built by gcc 10.2.1 20210110 (Debian 10.2.1-6)
instance/7418e88e7b7f4a40 2021/11/20 10:34:30 [notice] 1#1: OS: Linux 4.14.231-180.360.amzn2.x86_64
instance/7418e88e7b7f4a40 2021/11/20 10:34:30 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1024:4096
instance/7418e88e7b7f4a40 2021/11/20 10:34:30 [notice] 1#1: start worker processes
instance/7418e88e7b7f4a40 2021/11/20 10:34:30 [notice] 1#1: start worker process 30
instance/7418e88e7b7f4a40 2021/11/20 10:34:30 [notice] 1#1: start worker process 31
$ copilot svc package --env dev --name svc-rdws
# Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.
# SPDX-License-Identifier: Apache-2.0
AWSTemplateFormatVersion: 2010-09-09
Description: CloudFormation template that represents a request driven web service on AWS App Runner.
...
$ copilot svc status --env dev --name svc-rdws
Service Status
Status RUNNING
Last deployment
Updated At 48 minutes ago
Service ID app-1120-dev-svc-rdws/2479d8d636ac458baa5c1d8b71ae5546
Source app-1120/svc-rdws@sha256:8a8c0bf603c07d1443c73fa2cc396e8322dd14edc3ce464b9d42021a6c368d97
System Logs
2021-11-20T10:31:15Z [AppRunner] Service status is set to OPERATION_IN_PROGRESS.
2021-11-20T10:31:15Z [AppRunner] Starting to pull your application image.
2021-11-20T10:33:51Z [AppRunner] Successfully pulled image from ECR.
2021-11-20T10:34:04Z [AppRunner] Provisioning instances and deploying image.
2021-11-20T10:34:15Z [AppRunner] Performing health check on port '80'.
2021-11-20T10:34:43Z [AppRunner] Health check is successful. Routing traffic to application.
2021-11-20T10:35:10Z [AppRunner] Successfully routed incoming traffic to application.
2021-11-20T10:35:10Z [AppRunner] Service creation completed successfully.
2021-11-20T10:35:11Z [AppRunner] Service status is set to RUNNING.
2021-11-20T10:35:11Z [AppRunner] Finished pulling your application image.
Load Balanced Web Service
ECSをベースとして作成されるServiceタイプ。
Public Subnet内にECSサービスが配置され、ALB経由でリクエストを受けられる。
インターネット公開して、HTTPでやりとりするような普通のWebサーバーとして使える。
また、RDSに接続するなど、VPC依存のリソースと連携する場合に使える。
$ copilot svc init
Note: It's best to run this command in the root of your workspace.
Service type: Load Balanced Web Service
Service name: svc-lbws
Dockerfile: ./Dockerfile
✔ Wrote the manifest for service svc-lbws at copilot/svc-lbws/manifest.yml
Your manifest contains configurations like your container size and port (:80).
✔ Created ECR repositories for service svc-lbws..
Recommended follow-up actions:
- Update your manifest copilot/svc-lbws/manifest.yml to change the defaults.
- Run `copilot svc deploy --name svc-lbws --env test` to deploy your service to a test environment.
$ copilot svc deploy --name svc-lbws --env dev
Environment dev is already on the latest version v1.6.1, skip upgrade.
[+] Building 3.1s (8/8) FINISHED
...
✔ Proposing infrastructure changes for stack app-1120-dev-svc-lbws
- Creating the infrastructure for stack app-1120-dev-svc-lbws [create complete] [344.3s]
- Service discovery for your services to communicate within the VPC [create complete] [4.1s]
- Update your environment's shared resources [update complete] [184.5s]
- A security group for your load balancer allowing HTTP and HTTPS traffic [create complete] [7.8s]
- An Application Load Balancer to distribute public traffic to your services [create complete] [151.1s]
- An IAM Role for the Fargate agent to make AWS API calls on your behalf [create complete] [26.5s]
- A CloudWatch log group to hold your service logs [create complete] [4.1s]
- An ECS service to run and maintain your tasks in the environment cluster [create complete] [71.0s]
Deployments
Revision Rollout Desired Running Failed Pending
PRIMARY 1 [completed] 1 1 0 0
- A target group to connect the load balancer to your service [create complete] [1.5s]
- An ECS task definition to group your containers and run them on ECS [create complete] [4.3s]
- An IAM role to control permissions for the containers in your tasks [create complete] [26.5s]
✔ Deployed service svc-lbws.
Recommended follow-up action:
You can access your service at http://app-1-Publi-1PV6B964IK1PT-2080132374.ap-northeast-1.elb.amazonaws.com over the internet.
$ copilot svc deploy --name svc-lbws --env prod
...
$ copilot svc exec --env dev --name svc-lbws
Execute `/bin/sh` in container svc-lbws in task 97c222d578c74bd091d4c5065c15236c.
Starting session with SessionId: ecs-execute-command-076257a033672133c
# hostname
ip-10-0-0-69.ap-northeast-1.compute.internal
# exit
Exiting session with sessionId: ecs-execute-command-076257a033672133c.
Backend Service
ECSをベースとして作成されるServiceタイプ。
Private Subnet内にECSサービスが配置され、インターネットからアクセスできない形となる。
ECS ServiceDiscoveryにより、{svc_name}.{env_name}.{app_name}.local:{port}のようなサービス固有のDNS名でリクエスト可能となっている。
$ copilot svc init
Note: It's best to run this command in the root of your workspace.
Service type: Backend Service
Service name: svc-bs
Dockerfile: ./Dockerfile
✔ Wrote the manifest for service svc-bs at copilot/svc-bs/manifest.yml
Your manifest contains configurations like your container size and port (:80).
✔ Created ECR repositories for service svc-bs..
Recommended follow-up actions:
- Update your manifest copilot/svc-bs/manifest.yml to change the defaults.
- Run `copilot svc deploy --name svc-bs --env test` to deploy your service to a test environment.
$ copilot svc deploy --name svc-bs --env dev
Environment dev is already on the latest version v1.6.1, skip upgrade.
[+] Building 2.5s (8/8) FINISHED
...
✔ Proposing infrastructure changes for stack app-1120-dev-svc-bs
- Creating the infrastructure for stack app-1120-dev-svc-bs [create complete] [121.3s]
- Service discovery for your services to communicate within the VPC [create complete] [0.6s]
- Update your environment's shared resources [create complete] [0.0s]
- An IAM Role for the Fargate agent to make AWS API calls on your behalf [create complete] [26.1s]
- A CloudWatch log group to hold your service logs [create complete] [4.1s]
- An ECS service to run and maintain your tasks in the environment cluster [create complete] [68.8s]
Deployments
Revision Rollout Desired Running Failed Pending
PRIMARY 1 [completed] 1 1 0 0
- An ECS task definition to group your containers and run them on ECS [create complete] [3.7s]
- An IAM role to control permissions for the containers in your tasks [create complete] [26.1s]
✔ Deployed service svc-bs.
Recommended follow-up action:
You can access your service at svc-bs.dev.app-1120.local:80 with service discovery.
$ copilot svc deploy --name svc-bs --env prod
...
$ copilot svc exec --env dev --name svc-lbws
Execute `/bin/sh` in container svc-lbws in task 97c222d578c74bd091d4c5065c15236c.
Starting session with SessionId: ecs-execute-command-0cc4ec04b8a729d32
# curl 'svc-bs.dev.app-1120.local:80'
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>HELLO WORLD</title>
</head>
<body>
HELLO WORLD 1120 v1
</body>
# exit
Exiting session with sessionId: ecs-execute-command-0cc4ec04b8a729d32.
Worker Service
ECSをベースとして作成されるServiceタイプ。
今までのServiceとは違い、Pub/Sub形式となっている。
他のServiceなどからAmazon SNSに対してメッセージを送ると、Amazon SQSへと送られるようになっている。
そして、このServiceとして作成されてECS ServiceはSQSからメッセージを受信することで、任意の処理を行えるようになっている。
ただし、SNSのTopicはメッセージを送る側のServiceに紐付ける形になっている。
事前にTopicを作成していないと、Worker側をデプロイできない。
$ copilot svc init
Note: It's best to run this command in the root of your workspace.
Service type: Worker Service
Service name: svc-ws
Dockerfile: ./Dockerfile.worker
parse EXPOSE: no EXPOSE statements in Dockerfile ./Dockerfile.worker
No SNS topics are currently deployed in all environments. You can customize subscriptions in your manifest.
✔ Wrote the manifest for service svc-ws at copilot/svc-ws/manifest.yml
Your manifest contains configurations like your container size and port.
✔ Created ECR repositories for service svc-ws..
Recommended follow-up actions:
- Update your manifest copilot/svc-ws/manifest.yml to change the defaults.
- Run `copilot svc deploy --name svc-ws --env test` to deploy your service to a test environment.
$ copilot svc deploy --name svc-ws --env dev
Environment dev is already on the latest version v1.6.1, skip upgrade.
[+] Building 87.7s (11/11) FINISHED
...
✔ Proposing infrastructure changes for stack app-1120-dev-svc-ws
- Creating the infrastructure for stack app-1120-dev-svc-ws [create complete] [233.4s]
- Update your environment's shared resources [create complete] [4.4s]
- A KMS key to encrypt messages in your queues [create complete] [124.2s]
- An events SQS queue to buffer messages [create complete] [1.7s]
- An IAM Role for the Fargate agent to make AWS API calls on your behalf [create complete] [28.3s]
- A CloudWatch log group to hold your service logs [create complete] [4.8s]
- An ECS service to run and maintain your tasks in the environment cluster [create complete] [58.5s]
Deployments
Revision Rollout Desired Running Failed Pending
PRIMARY 1 [completed] 1 1 0 0
- An ECS task definition to group your containers and run them on ECS [create complete] [5.9s]
- An IAM role to control permissions for the containers in your tasks [create complete] [28.3s]
✔ Deployed service svc-ws.
Recommended follow-up action:
Update svc-ws's code to leverage the injected environment variable "COPILOT_QUEUE_URI".
In JavaScript you can write `const eventsQueueURI = process.env.COPILOT_QUEUE_URI`.
また、Worker Serviceを使用した際の、具体的な実装は以下のようになる。
メッセージを送る側のService実装（”Backend Service”など）
client.js
const express = require('express');
const { SNSClient, PublishCommand } = require('@aws-sdk/client-sns');
const app = express();
const port = 3000;
const wrap = (fn) => (...args) => fn(...args).catch(args[2]);
const client = new SNSClient({ region: 'ap-northeast-1' });
const { helloTopic } = JSON.parse(process.env.COPILOT_SNS_TOPIC_ARNS);
app.get('/', (req, res) => {
res.send('Hello World!');
});
app.get('/send', wrap(async (req, res) => {
const out = await client.send(new PublishCommand({
Message: 'HELLO',
TopicArn: helloTopic,
}));
res.json({
helloTopic,
out,
});
}));
app.use((err, req, res, next) => {
if (err) {
res.status(500);
res.send(err);
} else {
res.send('OK');
}
});
app.listen(port, () => {
console.log(`Example app listening at http://localhost:${port}`);
});
メッセージを受け取る側のService実装（”Worker Service”）
worker.js
const { SQSClient, ReceiveMessageCommand, DeleteMessageCommand } = require('@aws-sdk/client-sqs');
const queueUrl = process.env.COPILOT_QUEUE_URI;
const client = new SQSClient({ region: 'ap-northeast-1' });
(async () => {
console.log('queueUrl', queueUrl);
while (true) {
try {
const out = await client.send(new ReceiveMessageCommand({
QueueUrl: queueUrl,
WaitTimeSeconds: 10,
}))
console.log('out', JSON.stringify(out));
if (out.Messages === undefined || out.Messages.length === 0) {
continue;
}
for (const message of out.Messages) {
console.log('message', JSON.stringify(message));
}
await client.send(new DeleteMessageCommand({
QueueUrl: queueUrl,
ReceiptHandle: out.Messages[0].ReceiptHandle,
}))
} catch (err) {
console.error(err);
}
}
})();
まとめ
各Serviceタイプの違いや、全Serviceタイプを組み合わせたときの全体像が何となくつかめてきた。
これらをゼロベースでCopilotなしに構築しようとしたら、相当面倒くさいだろう。。。
それをCopilotのCLIからコマンド叩くだけで簡単に作れてしまうのはとても良い。
さらにAmplifyもCopilotで管理できたら、色々と便利になりそう。
まだ、Job・Pipelineは触れていないし、各Serviceの細かい使い方も追って確認していきたい。
全体図
（Zenn側で変換される画像の解像度をもうちょっと上げてほしい感）
Copilotシリーズ
Discussion