Zenn
😽

ECS サービスディスカバリを利用してサービス間通信をTerrafromで実現

2024/10/13に公開
AWS
Terraform
DNS
Route53
ECS
tech

はじめに

以前、コンテナ間通信の記事を作成しました。
https://zenn.dev/osachi/articles/481f64950a9a65

今回はコンテナ間ではなくて、サービス間の通信を実現します。
インターネットを介さずに、プライベートネットワーク内部でのサービス間通信を実現します。

サービスディスカバリ（サービス検出）とは

AWS Cloud Map というサービスを利用して、同じVPC内のECCServiceのタスクに対して名前解決をしてくれるサービスです。 ECS Serviceに対して設定していきます。
https://docs.aws.amazon.com/ja_jp/AmazonECS/latest/developerguide/service-discovery.html

AWS構成図

今回、実現する構成図です。
2つのサービスを作成します。

backend_service

api.ecs-navi.siteに対するリクエストを受け取って、phpinfo() を返却します。　※前回同様

nginx_proxy_service

ecs-navi.siteに対するリクエストを受け取って、サービスディスカバリを利用してbackend_serviceにリクエストします。
つまり、api.ecs-navi.siteにリクエストされても、ecs-navi.siteにリクエストされても、backend_serviceのphpinfo() を返却します。

ユーザー
  |
  | HTTPS (443)
  v
[Route53]
  |
  v
[Application Load Balancer (ALB)] -- パブリックサブネット
  |                        |
  | Host: ecs-navi.site    | Host: api.ecs-navi.site
  v                        v
-------------------------------- -- プライベートサブネット
[nginx_proxy_tg]          [backend_tg]
  |                        |
  v                        v
[nginx_proxy_service]     [backend_service]
  (プロキシnginx)          (nginx + api)
     |                         |
     | HTTP (80)               |
     |------------------------>|
          サービスディスカバリ
          (backend.ecs-navi.local)

コンテナイメージを作成

backend_service

こちらは、以前の記事と同様です。

Dockerfile

FROM php:8.2-fpm

WORKDIR /var/www

COPY ./docker/api/php.ini /usr/local/etc/php/

RUN apt-get update && apt-get -y upgrade
RUN apt-get install -y zip \ 
    unzip \
    vim

WORKDIR /var/www/src/public
RUN touch index.php
RUN chmod 777 index.php
RUN echo "<?php phpinfo();" > index.php

php.ini

[Date]
date.timezone = "Asia/Tokyo"
[mbstring]
mbstring.internal_encoding = "UTF-8"
mbstring.language = "Japanese"

memory_limit = 256M

nginx_proxy_service

Dockerfile

FROM nginx:stable-alpine
COPY ./docker/nginx/default.conf /etc/nginx/conf.d/default.conf

default.conf

今回のポイントの１つは、nginx_proxyコンテナ内部で、passを http://backend.ecs-navi.local:80 に設定しています。後に記述しますが、サービスディスカバリによって、backend_serviceへの名前解決をしてくれるため、nginx_proxyコンテナ内からVPC内部のプライベートIPへリクエストが可能になります。

server {
    listen 80;

    location / {
        proxy_pass http://backend.ecs-navi.local:80;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
}

ECRへPUSH

ECRのリポジトリへPUSHしてください。
<account-id>.dkr.ecr.ap-northeast-1.amazonaws.com/ecs-navi-nginx:latest
<account-id>.dkr.ecr.ap-northeast-1.amazonaws.com/ecs-navi-nginx-proxy:latest

TerraformでECSのリソース作成

NW

NWは以前と同様です.


// VPC
resource "aws_vpc" "ecs_navi_vpc" {
  cidr_block = "10.0.0.0/16"
  enable_dns_support = true
  enable_dns_hostnames = true

  tags = {
    Name = "${var.service_name}-vpc"
  }
}

// Subnet
resource "aws_subnet" "ecs_navi_subnet_public_1a" {
  vpc_id = aws_vpc.ecs_navi_vpc.id
  cidr_block = "10.0.1.0/24"
  availability_zone = "ap-northeast-1a"
  map_public_ip_on_launch = true

  tags = {
    Name = "${var.service_name}-subnet-public-1a"
  }
}

resource "aws_subnet" "ecs_navi_subnet_public_1c" {
  vpc_id = aws_vpc.ecs_navi_vpc.id
  cidr_block = "10.0.2.0/24"
  availability_zone = "ap-northeast-1c"
  map_public_ip_on_launch = true
  
  tags = {
    Name = "${var.service_name}-subnet-public-1c"
  }
}

resource "aws_subnet" "ecs_navi_subnet_private_1a" {
  vpc_id = aws_vpc.ecs_navi_vpc.id
  cidr_block = "10.0.10.0/24"
  availability_zone = "ap-northeast-1a"
  map_public_ip_on_launch = false
  
  tags = {
    Name = "${var.service_name}-subnet-private-1a"
  }
}

resource "aws_subnet" "ecs_navi_subnet_private_1c" {
  vpc_id = aws_vpc.ecs_navi_vpc.id
  cidr_block = "10.0.11.0/24"
  availability_zone = "ap-northeast-1c"
  map_public_ip_on_launch = false
  
  tags = {
    Name = "${var.service_name}-subnet-private-1c"
  }
}

// Internet Gateway
resource "aws_internet_gateway" "ecs_navi_igw" {
  vpc_id = aws_vpc.ecs_navi_vpc.id 
}

// Route Table
resource "aws_route_table" "ecs_navi_route_table" {
  vpc_id = aws_vpc.ecs_navi_vpc.id

  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.ecs_navi_igw.id
  }
}

// Route Table Association
resource "aws_route_table_association" "ecs_navi_subnet_public_1a_association" {
  subnet_id = aws_subnet.ecs_navi_subnet_public_1a.id
  route_table_id = aws_route_table.ecs_navi_route_table.id
}

resource "aws_route_table_association" "ecs_navi_subnet_public_1c_association" {
  subnet_id = aws_subnet.ecs_navi_subnet_public_1c.id
  route_table_id = aws_route_table.ecs_navi_route_table.id
}

// eip
resource "aws_eip" "ecs_navi_eip" {
  domain = "vpc"
}

// Nat Gateway
resource "aws_nat_gateway" "ecs_navi_nat_gateway" {
  subnet_id = aws_subnet.ecs_navi_subnet_public_1a.id
  allocation_id = aws_eip.ecs_navi_eip.id
}

// Route Table
resource "aws_route_table" "ecs_navi_route_table_private" {
  vpc_id = aws_vpc.ecs_navi_vpc.id

  route {
    cidr_block = "0.0.0.0/0"
    nat_gateway_id = aws_nat_gateway.ecs_navi_nat_gateway.id
  }
}

// Route Table Association
resource "aws_route_table_association" "ecs_navi_subnet_private_1a_association" {
  subnet_id = aws_subnet.ecs_navi_subnet_private_1a.id
  route_table_id = aws_route_table.ecs_navi_route_table_private.id
}

resource "aws_route_table_association" "ecs_navi_subnet_private_1c_association" {
  subnet_id = aws_subnet.ecs_navi_subnet_private_1c.id
  route_table_id = aws_route_table.ecs_navi_route_table_private.id
}

POLICY

今回重要な点は、タスクにsevice discoveryをするための権限が必要であることです。この権限がないと名前入解決を出来ません。


// IAM
resource "aws_iam_role" "ecs_navi_ecs_task_execution_role" {
  name = "${var.service_name}-ecs-task-execution-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = {
          Service = "ecs-tasks.amazonaws.com"
        }
      }
    ]
  })

  tags = {
    Name = "${var.service_name}-ecs-task-execution-role"
  }
}

resource "aws_iam_role_policy" "ecs_navi_ecs_task_execution_policy" {
  name = "${var.service_name}-ecs-task-execution-policy"
  role = aws_iam_role.ecs_navi_ecs_task_execution_role.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = [
          "ecr:GetDownloadUrlForLayer",
          "ecr:BatchGetImage",
          "ecr:GetAuthorizationToken",
          "ecr:BatchCheckLayerAvailability",
          "ecr:GetDownloadUrlForLayer",
        ]
        Resource = "*"
      },
      {
        Effect = "Allow"
        Action = [
          "logs:CreateLogStream",
          "logs:PutLogEvents",
          "logs:CreateLogGroup",
          "logs:DescribeLogGroups",
          "logs:DescribeLogStreams",
        ]
        Resource = "*"
      },
      {
        Effect = "Allow"
        Action = [
          "ecs:DescribeTasks",
          "ecs:UpdateContainerInstancesState",
          "ecs:SubmitContainerStateChange"
        ]
        Resource = "*"
      },
      // sevice discovery権限
      {
        "Effect": "Allow",
        "Action": [
            "servicediscovery:RegisterInstance",
            "servicediscovery:DeregisterInstance",
            "servicediscovery:ListServices",
            "servicediscovery:GetService",
            "servicediscovery:GetInstancesHealthStatus",
            "servicediscovery:DiscoverInstances"
        ],
        "Resource": "*"
      },
      // exec権限
      {
        "Effect": "Allow",
        "Action": [
            "ssmmessages:CreateControlChannel",
            "ssmmessages:CreateDataChannel",
            "ssmmessages:OpenControlChannel",
            "ssmmessages:OpenDataChannel"
        ],
        "Resource": "*"
      }
    ]
  })
}

セキュリティグループ

3つのセキュリティグループを用意しました。

  • ロードバランサーのSG
  • backend_serviceのSG
  • nginx_proxy_serviceのSG

backend_serviceのSGのインバウンドルールには、ロードバランサーだけではなくて、nginx_proxy_serviceからのアクセスも許可してあげます。

// セキュリティグループ
resource "aws_security_group" "ecs_navi_sg" {
  vpc_id = aws_vpc.ecs_navi_vpc.id

  ingress {
    from_port = 443
    to_port = 443
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
    Name = "${var.service_name}-sg"
  }
}

// セキュリティグループbackendサービス
resource "aws_security_group" "ecs_navi_backend_sg" {
  vpc_id = aws_vpc.ecs_navi_vpc.id

  ingress {
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    security_groups = [aws_security_group.ecs_navi_sg.id, aws_security_group.nginx_proxy_sg.id]
  }

  ingress {
    from_port   = 3000
    to_port     = 3000
    protocol    = "tcp"
    security_groups = [aws_security_group.ecs_navi_sg.id]
  }


  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
    Name = "${var.service_name}-backend-sg"
  }
}

// セキュリティグループproxyサービス
resource "aws_security_group" "nginx_proxy_sg" {
  vpc_id = aws_vpc.ecs_navi_vpc.id

  ingress {
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    security_groups = [aws_security_group.ecs_navi_sg.id]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
    Name = "${var.service_name}-proxy-nginx-sg"
  }
}

ROUTE53

※今回は、ホストゾーンのリソース作成・ACMのリソース作成は省略しました

resource "aws_route53_record" "ecs_navi_site" {
  zone_id = aws_route53_zone.main.zone_id 
  name    = "ecs-navi.site" 
  type    = "A"

  alias {
    name                   = aws_lb.ecs_navi_lb.dns_name
    zone_id                = aws_lb.ecs_navi_lb.zone_id
    evaluate_target_health = false
  }
}

resource "aws_route53_record" "api_ecs_navi_site" {
  zone_id = aws_route53_zone.main.zone_id
  name    = "api.ecs-navi.site"
  type    = "A"

  alias {
    name                   = aws_lb.ecs_navi_lb.dns_name 
    zone_id                = aws_lb.ecs_navi_lb.zone_id
    evaluate_target_health = false
  }
}

ロードバランサー


// ロードバランサー
resource "aws_lb" "ecs_navi_lb" {
  name               = "${var.service_name}-lb"
  internal           = false
  load_balancer_type = "application"
  security_groups    = [aws_security_group.ecs_navi_sg.id]
  subnets            = [aws_subnet.ecs_navi_subnet_public_1a.id, aws_subnet.ecs_navi_subnet_public_1c.id]

  enable_deletion_protection = false
}

resource "aws_lb_target_group" "backend_tg" {
  name     = "${var.service_name}-backend-tg"
  port     = 80
  protocol = "HTTP"
  vpc_id   = aws_vpc.ecs_navi_vpc.id
  target_type = "ip"

  health_check {
    path                = "/"
    interval            = 300
    timeout             = 120
    healthy_threshold   = 5
    unhealthy_threshold = 10
    matcher             = "200"
  }
}

resource "aws_lb_target_group" "nginx_proxy_tg" {
  name     = "${var.service_name}-nginx-proxy-tg"
  port     = 3000
  protocol = "HTTP"
  vpc_id   = aws_vpc.ecs_navi_vpc.id
  target_type = "ip"

  health_check {
    path                = "/"
    interval            = 300
    timeout             = 120
    healthy_threshold   = 5
    unhealthy_threshold = 10
    matcher             = "200"
  }
}

resource "aws_lb_listener" "https" {
  load_balancer_arn = aws_lb.ecs_navi_lb.arn
  port              = "443"
  protocol          = "HTTPS"
  ssl_policy        = "ELBSecurityPolicy-2016-08"
  certificate_arn   = aws_acm_certificate.ecs_navi_cert.arn

  default_action {
    type             = "fixed-response"
    fixed_response {
      content_type = "text/plain"
      message_body = "404 Not Found"
      status_code  = "404"
    }
  }
}

# ecs-navi.site Rule
resource "aws_lb_listener_rule" "ecs_navi_site_https_rule" {
  listener_arn = aws_lb_listener.https.arn
  priority     = 100

  action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.nginx_proxy_tg.arn
  }

  condition {
    host_header {
      values = ["ecs-navi.site"]
    }
  }
}

# api.ecs-navi.site Rule
resource "aws_lb_listener_rule" "api_ecs_navi_site_https_rule" {
  listener_arn = aws_lb_listener.https.arn
  priority     = 200

  action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.backend_tg.arn
  }

  condition {
    host_header {
      values = ["api.ecs-navi.site"]
    }
  }
}

AWSCloudMap

今回の重要なリソースの１つであるAWSCloudMapのリソースを作成します。
これらリソースを作成することによって、

  • Route53のプライベートタイプに ecs-navi.local というホストゾーンが作成されます。
  • AWSCloudMapにecs-navi.local という名前空間が作成されます。
resource "aws_service_discovery_private_dns_namespace" "ecs_navi_namespace" {
  name        = "ecs-navi.local"
  vpc         = aws_vpc.ecs_navi_vpc.id
  description = "Private DNS namespace for ECS Navi services"
}

resource "aws_service_discovery_service" "ecs_navi_backend_service_discovery" {
  name = "backend"
  dns_config {
    namespace_id = aws_service_discovery_private_dns_namespace.ecs_navi_namespace.id
    dns_records {
      type = "A"
      ttl  = 60
    }
  }
  health_check_custom_config {
    failure_threshold = 1
  }
}

resource "aws_service_discovery_service" "nginx_proxy_service_discovery" {
  name = "proxy-nginx"
  dns_config {
    namespace_id = aws_service_discovery_private_dns_namespace.ecs_navi_namespace.id
    dns_records {
      type = "A"
      ttl  = 60
    }
  }
  health_check_custom_config {
    failure_threshold = 1
  }
}

ECS

ポイントは、ECS Serviceにservice_registriesを設定したことです。
これによって、先程のAWS Cloud Mapと連携して、名前解決をしてくれます。



// ECS Cluster
resource "aws_ecs_cluster" "ecs_navi_cluster" {
  name = "${var.service_name}-cluster"
}

// ECS Service
resource "aws_ecs_service" "backend_service" {
  name            = "${var.service_name}-backend-service"
  cluster         = aws_ecs_cluster.ecs_navi_cluster.id
  task_definition = aws_ecs_task_definition.ecs_navi_nginx_task_definition.arn
  desired_count   = 1
  launch_type     = "FARGATE"

  network_configuration {
    subnets         = [aws_subnet.ecs_navi_subnet_private_1a.id, aws_subnet.ecs_navi_subnet_private_1c.id]
    security_groups = [aws_security_group.ecs_navi_backend_sg.id]
  }

  load_balancer {
    target_group_arn = aws_lb_target_group.backend_tg.arn
    container_name   = "nginx"
    container_port   = 80
  }

  service_registries {
    registry_arn = aws_service_discovery_service.ecs_navi_backend_service_discovery.arn
  }

  enable_execute_command = true
}


// ECS タスク定義
resource "aws_ecs_task_definition" "ecs_navi_nginx_task_definition" {
  family = "${var.service_name}-nginx-task"
  requires_compatibilities = ["FARGATE"]
  network_mode = "awsvpc"
  cpu                      = 256
  memory                   = 512

  container_definitions = jsonencode([
    {
      name = "nginx"
      image     = "<account-id>.dkr.ecr.ap-northeast-1.amazonaws.com/ecs-navi-nginx:latest"
      portMappings = [
        {
          containerPort = 80
          hostPort = 80
        }
      ]
      logConfiguration = {
        logDriver = "awslogs"
        options = {
          awslogs-group         = "/ecs/nginx"
          awslogs-region        = "ap-northeast-1"
          awslogs-stream-prefix = "ecs-nginx"
        }
      }
    },
    {
      name = "api"
      image     = "<account-id>.dkr.ecr.ap-northeast-1.amazonaws.com/ecs-navi-api:latest"
      logConfiguration = {
        logDriver = "awslogs"
        options = {
          awslogs-group         = "/ecs/api"
          awslogs-region        = "ap-northeast-1"
          awslogs-stream-prefix = "ecs-api"
        }
      }
    }
  ])

  task_role_arn = aws_iam_role.ecs_navi_ecs_task_execution_role.arn
  execution_role_arn = aws_iam_role.ecs_navi_ecs_task_execution_role.arn
}

# CloudWatch Logs
resource "aws_cloudwatch_log_group" "nginx" {
  name              = "/ecs/nginx"
  retention_in_days = 7
}

# CloudWatch Logs
resource "aws_cloudwatch_log_group" "api" {
  name              = "/ecs/api"
  retention_in_days = 7
}



resource "aws_ecs_task_definition" "nginx_proxy_task_definition" {
  family                   = "${var.service_name}-nginx-proxy-task"
  requires_compatibilities = ["FARGATE"]
  network_mode             = "awsvpc"
  cpu                      = 256
  memory                   = 512

  container_definitions = jsonencode([
    {
      name      = "nginx"
      image     = "<account-id>.dkr.ecr.ap-northeast-1.amazonaws.com/ecs-navi-nginx-proxy:latest"
      portMappings = [
        {
          containerPort = 80
          hostPort      = 80
          protocol      = "tcp"
        }
      ]
    }
  ])

  execution_role_arn = aws_iam_role.ecs_navi_ecs_task_execution_role.arn
  task_role_arn      = aws_iam_role.ecs_navi_ecs_task_execution_role.arn
}

resource "aws_ecs_service" "nginx_proxy_service" {
  name            = "${var.service_name}-nginx-proxy-service"
  cluster         = aws_ecs_cluster.ecs_navi_cluster.id
  task_definition = aws_ecs_task_definition.nginx_proxy_task_definition.arn
  desired_count   = 1
  launch_type     = "FARGATE"

  network_configuration {
    subnets         = [aws_subnet.ecs_navi_subnet_private_1a.id, aws_subnet.ecs_navi_subnet_private_1c.id]
    security_groups = [aws_security_group.nginx_proxy_sg.id]
  }

  load_balancer {
    target_group_arn = aws_lb_target_group.nginx_proxy_tg.arn
    container_name   = "nginx"
    container_port   = 80
  }

  service_registries {
    registry_arn = aws_service_discovery_service.nginx_proxy_service_discovery.arn
  }

  enable_execute_command = true
}

確認

アクセス

api.ecs-navi.siteへのアクセスも、ecs-navi.siteへのアクセスも phpinfoの情報が返却されることが確認できます。

Aレコード作成

Route53の ecs-navi.local ホストゾーンでAレコードを確認してみます。

サービスディスカバリで生成されたAレコードが登録されています。

コンテナから名前解決

nginx-proxyのコンテナにログインしてみて、名前解決されているのかを確認してみます。
コンテナログイン方法はこちらです。

コマンド

 aws ecs execute-command \
    --profile private \
    --cluster ecs-navi-cluster \
    --task <task-id> \
    --container nginx \
    --interactive \
    --command "sh"

コンテナ内部からnsloolup

コンテナ内部から　nslookup backend.ecs-navi.local　を実行します。
先ほどのRoute53で作成されたいIPが出力されました。

Non-authoritative answer:
Name:	backend.ecs-navi.local
Address: 10.0.11.82

サービスディスカバリの自動検出機能

試しにbackend_serviceのタスク数を３つにしてみます。
その後に、Route53・名前解決を確認してみます。

Route53でAレコードが３つに増えています。

コンテナ内部からnsloolup

同様にnslookup backend.ecs-navi.local　を実行すると名前解決されていることが分かります。

Non-authoritative answer:
Name:	backend.ecs-navi.local
Address: 10.0.10.10
Name:	backend.ecs-navi.local
Address: 10.0.11.82
Name:	backend.ecs-navi.local
Address: 10.0.11.120

Discussion