🦔

委任したAccountでAWS Configを観測 by Terraform

18 min read

概要

またAWS Landing Zone環境構築の続き
今回はTerraformを用いてマルチアカウント環境でAWS Configを設定する

ポイントとしては以下

  1. Log Archive Accountに組織Aggregatorを委任
  2. 全アカウントのConfig計測データをLog Archive AccountのS3に集約
    1. 各アカウントのConfig設定を一つのterraformで実施
  3. 組織内全アカウントにConfig Ruleを適用

イメージとしては以下のような感じ

ディレクトリ構成

今回こんな感じで作成した
(やはりTerraform歴浅いのでまだまだ実力不足感を感じる

$ tree
.
├── backend.tf
├── main.tf
├── modules
│   └── config
│       ├── aggregator
│       │   ├── main.tf
│       │   ├── outputs.tf
│       │   └── variables.tf
│       ├── each-config
│       │   ├── main.tf
│       │   ├── outputs.tf
│       │   └── variables.tf
│       └── rules
│           ├── main.tf
│           ├── outputs.tf
│           └── variables.tf
├── provider.tf
└── versions.tf

5 directories, 13 files

やったこと

それぞれやったことのポイントをメモしておく

各アカウント情報の設定

各AWSアカウントにConfig設定するために
providerのassume_roleにrole_arnを指定する

main.tfでこのproviderのaliasを指定することで
対象のAWSアカウントに対してリソースを作成したりできる🤓

roleは前回のSSO記事で生成されたAdministratorAccessの付与されたrole使ってる

./provider.tf
provider "aws" {
  region  = "ap-northeast-1"
}

provider "aws" {
  alias = "log-archive-production"

  region  = "ap-northeast-1"
  assume_role {
    role_arn = "arn:aws:iam::${aws_organizations_account.log_archive_production_account.id}:role/OrganizationAccountAccessRole"
  }
}

provider "aws" {
  alias = "ucwork-production-account"

  region  = "ap-northeast-1"
  assume_role {
    role_arn = "arn:aws:iam::${aws_organizations_account.ucwork_production_account.id}:role/OrganizationAccountAccessRole"
  }
}

# ...アカウントの分だけproviderを用意する

各リソース作成指示

各種moduleやresourceの作成
Organizational Unit(OU), Account, SSOは
コメントに記載した別記事参照くだされ

./main.tf
# -----------------------------------------------------------
# Set up organizations
# -----------------------------------------------------------

// この辺参照ください
https://zenn.dev/ucwork/articles/abebfe4003a7ee

// OUとAccountを管理してる場所

# -----------------------------------------------------------
# Set up aggregator and each config in all account
# -----------------------------------------------------------
// terraformのresourceないみたいなのでawsコマンドで実行
// 登録済みの場合エラーになるのでon_failure = continueをセット
resource "null_resource" "config_delegated" {
  provisioner "local-exec" {
    command = "aws organizations register-delegated-administrator --account-id ${aws_organizations_account.log_archive_production_account.id} --service-principal config.amazonaws.com"
    on_failure = continue
  }
}

resource "null_resource" "config_multi_setup_delegated" {
  provisioner "local-exec" {
    command = "aws organizations register-delegated-administrator --account-id ${aws_organizations_account.log_archive_production_account.id} --service-principal config-multiaccountsetup.amazonaws.com"
    on_failure = continue
  }
  depends_on = [ null_resource.config_delegated ]
}

module "config_aggregator" {
  source = "./modules/config/aggregator"

  aggregator_account_id = aws_organizations_account.log_archive_production_account.id
  aggregator_s3_region = "ap-northeast-1"

  providers = {
    aws = aws.log-archive-production
  }
}

// config_aggregatorで作成したS3使いたいので依存関係(depends_on)指定
module "config_management" {
  source = "./modules/config/each-config"

  bucket_arn = module.config_aggregator.config_s3_arn
  bucket_id = module.config_aggregator.config_s3_id

  depends_on = [
    module.config_aggregator
  ]
}

// provider.tfで指定したaliasで各アカウントにresource作成
module "config_ucwork_production" {
  source = "./modules/config/each-config"

  bucket_arn = module.config_aggregator.config_s3_arn
  bucket_id = module.config_aggregator.config_s3_id

  providers = {
    aws = aws.ucwork-production-account
  }
  depends_on = [
    module.config_aggregator
  ]
}

# Config設定するアカウントの分だけ上記each-configのmoduleを追記する!
// moduleのfor_each使いたかったけど、providersのaliasにeach.value指定するとエラーで怒られたので愚直に全アカウント分書いた・・・

# -----------------------------------------------------------
# Set up organization config rules
# -----------------------------------------------------------
// organizationに紐づくアカウント全部にルール作成
// 各アカウントのconfig resourceないと怒られるのでdepends_onを指定
module "config_rules" {
  source = "./modules/config/rules"

  depends_on = [
    module.config_management,
    module.config_shared_services_production,
    module.config_ucwork_production,
    module.config_ucwork_sdlc
  ]
}

# -----------------------------------------------------------
# Set up Single Sign-On (SSO)
# -----------------------------------------------------------

// この辺見てね
https://zenn.dev/ucwork/articles/7f30488b6d2bbf

// SSOでマルチアカウントアクセスするための設定まわりが書いてある

Aggregatorアカウントの設定

./modules/config/aggregator/main.tf
# logging s3
resource "aws_s3_bucket" "config_bucket" {
  bucket = "config-bucket-${var.aggregator_account_id}-${var.aggregator_s3_region}"
  acl    = "private"

  server_side_encryption_configuration {
    rule {
      apply_server_side_encryption_by_default {
        sse_algorithm = "AES256"
      }
    }
  }
}

resource "aws_s3_bucket_policy" "config_logging_policy" {
  bucket = aws_s3_bucket.config_bucket.id
  policy = <<POLICY
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AWSConfigBucketPermissionsCheck",
      "Effect": "Allow",
      "Principal": {
        "Service": [
         "config.amazonaws.com"
        ]
      },
      "Action": "s3:GetBucketAcl",
      "Resource": "${aws_s3_bucket.config_bucket.arn}"
    },
    {
      "Sid": "AWSConfigBucketExistenceCheck",
      "Effect": "Allow",
      "Principal": {
        "Service": [
          "config.amazonaws.com"
        ]
      },
      "Action": "s3:ListBucket",
      "Resource": "${aws_s3_bucket.config_bucket.arn}"
    },
    {
      "Sid": "AWSConfigBucketDelivery",
      "Effect": "Allow",
      "Principal": {
        "Service": [
         "config.amazonaws.com"
        ]
      },
      "Action": "s3:PutObject",
      "Resource": "${aws_s3_bucket.config_bucket.arn}/AWSLogs/*/Config/*",
      "Condition": {
        "StringEquals": {
          "s3:x-amz-acl": "bucket-owner-full-control"
        }
      }
    }
  ]
}
POLICY
}

# iam role
resource "aws_iam_role" "config_role" {
  name = "config_aggregator_role"

  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": "config.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF
}

resource "aws_iam_policy" "config_org_policy" {
  path        = "/"
  description = "IAM Policy for AWS Config"
  name        = "ConfigPolicy"

  policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "config:GetOrganizationConfigRuleDetailedStatus",
        "config:Put*",
        "iam:GetPasswordPolicy",
        "organizations:ListAccounts",
        "organizations:DescribeOrganization",
        "organizations:ListAWSServiceAccessForOrganization",
        "organization:EnableAWSServiceAccess"
      ],
      "Effect": "Allow",
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": ["s3:PutObject"],
      "Resource": ["${aws_s3_bucket.config_bucket.arn}/AWSLogs/${var.aggregator_account_id}/*"],
      "Condition": {
        "StringLike": {
          "s3:x-amz-acl": "bucket-owner-full-control"
        }
      }
    },
    {
      "Effect": "Allow",
      "Action": ["s3:GetBucketAcl"],
      "Resource": "${aws_s3_bucket.config_bucket.arn}"
    }
  ]
}
EOF
}

resource "aws_iam_role_policy_attachment" "config_org_policy_attach" {
  role       = aws_iam_role.config_role.name
  policy_arn = aws_iam_policy.config_org_policy.arn
}

resource "aws_iam_role_policy_attachment" "config_policy_attach" {
  role       = aws_iam_role.config_role.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSConfigRole"
}

resource "aws_iam_role_policy_attachment" "read_only_policy_attach" {
  role       = aws_iam_role.config_role.name
  policy_arn = "arn:aws:iam::aws:policy/ReadOnlyAccess"
}

# -----------------------------------------------------------
# set up the aggregator account Config
# -----------------------------------------------------------
resource "aws_config_configuration_aggregator" "organization" {
  name = "organization-aggregator"

  organization_aggregation_source {
    all_regions = true
    role_arn    = aws_iam_role.config_role.arn
  }
}

resource "aws_config_configuration_recorder" "config_recorder" {
  role_arn = aws_iam_role.config_role.arn

  recording_group {
    all_supported                 = true
    include_global_resource_types = true
  }
}

# Delivery channel resource and bucket location to specify configuration history location.
resource "aws_config_delivery_channel" "config_channel" {
  s3_bucket_name = aws_s3_bucket.config_bucket.id
  depends_on = [aws_config_configuration_recorder.config_recorder]
}

resource "aws_config_configuration_recorder_status" "config_recorder_status" {
  name       = aws_config_configuration_recorder.config_recorder.name
  is_enabled = true
  depends_on = [aws_config_delivery_channel.config_channel]
}
./modules/config/aggregator/variables.tf
variable "aggregator_account_id" {
  type        = string
  default     = ""
  description = "config aggregator account id"
}

variable "aggregator_s3_region" {
  type        = string
  default     = ""
  description = "config aggregator s3 region"
}
./modules/config/aggregator/outputs.tf
output "config_s3_arn" {
    value = aws_s3_bucket.config_bucket.arn
    description = "AWS Config Aggregator s3 bucket arn"
}

output "config_s3_id" {
    value = aws_s3_bucket.config_bucket.id
    description = "AWS Config Aggregator s3 bucket name(id)"
}

各アカウントのConfig設定

./modules/config/each-config/main.tf
resource "aws_iam_role" "config_role" {
  name = "ConfigRecorderRole"

  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": "config.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF
}

resource "aws_iam_policy" "s3_config_org_policy" {
  path        = "/"
  description = "S3ConfigOrganizationPolicy"

  policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:PutObject"],
      "Resource": ["${var.bucket_arn}/AWSLogs/*/Config/*"],
      "Condition": {
        "StringLike": {
          "s3:x-amz-acl": "bucket-owner-full-control"
        }
      }
    },
    {
      "Effect": "Allow",
      "Action": ["s3:GetBucketAcl"],
      "Resource": "${var.bucket_arn}"
    }
  ]
}
EOF
}

resource "aws_iam_role_policy_attachment" "config_s3_policy_attach" {
  role       = aws_iam_role.config_role.name
  policy_arn = aws_iam_policy.s3_config_org_policy.arn
}

resource "aws_iam_role_policy_attachment" "read_only_attachment" {
  role       = aws_iam_role.config_role.name
  policy_arn = "arn:aws:iam::aws:policy/ReadOnlyAccess"
}

resource "aws_iam_role_policy_attachment" "config_attachment" {
  role       = aws_iam_role.config_role.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSConfigRole"
}

# -----------------------------------------------------------
# set up the Config
# -----------------------------------------------------------
resource "aws_config_configuration_recorder" "config_recorder" {
  role_arn = aws_iam_role.config_role.arn
}

resource "aws_config_delivery_channel" "config_channel" {
  s3_bucket_name = var.bucket_id
  depends_on = [aws_config_configuration_recorder.config_recorder]
}

resource "aws_config_configuration_recorder_status" "config_recorder_status" {
  name       = aws_config_configuration_recorder.config_recorder.name
  is_enabled = true
  depends_on = [aws_config_delivery_channel.config_channel]
}
./modules/config/each-config/variables.tf
variable "bucket_arn" {
  type        = string
  default     = ""
  description = "organization config s3 bucket arn"
}

variable "bucket_id" {
  type        = string
  default     = ""
  description = "organization config s3 bucket name(id)"
}

config ruleの設定

./modules/config/rules/main.tf
# AWS Config Rule that manages IAM Password Policy
resource "aws_config_organization_managed_rule" "iam_policy_organization_config_rule" {
  input_parameters  = <<EOF
    {
      "RequireUppercaseCharacters": "true",
      "RequireLowercaseCharacters": "true",
      "RequireSymbols": "true",
      "RequireNumbers": "true",
      "MinimumPasswordLength": "9",
      "PasswordReusePrevention": "5",
      "MaxPasswordAge": "90"
    }
  EOF

  name              = "iam-password-policy"
  rule_identifier   = "IAM_PASSWORD_POLICY"
}

# AWS Config Rule that manages IAM Root Access Keys to see if they exist
resource "aws_config_organization_managed_rule" "iam_root_access_key_organization_config_rule" {
  name              = "iam-root-access-key-check"
  rule_identifier   = "IAM_ROOT_ACCESS_KEY_CHECK"
}

# AWS Config Rule that checks whether your AWS account is enabled to use multi-factor authentication (MFA)
# hardware device to sign in with root credentials.
resource "aws_config_organization_managed_rule" "root_hardware_mfa_organization_config_rule" {
  name              = "root-hardware-mfa"
  rule_identifier   = "ROOT_ACCOUNT_HARDWARE_MFA_ENABLED"
}

# AWS Config Rule that checks whether users of your AWS account require a multi-factor authentication (MFA)
# device to sign in with root credentials.
resource "aws_config_organization_managed_rule" "root_account_mfa_organization_config_rules" {
  name              = "root-account-mfa-enabled"
  rule_identifier   = "ROOT_ACCOUNT_MFA_ENABLED"
}


# AWS Config Rule that checks whether the required public access block settings are configured from account level.
# The rule is only NON_COMPLIANT when the fields set below do not match the corresponding fields in the configuration
# item.
resource "aws_config_organization_managed_rule" "s3_public_access_organization_config_rules" {
  name              = "s3-account-level-public-access-blocks"
  rule_identifier   = "S3_ACCOUNT_LEVEL_PUBLIC_ACCESS_BLOCKS"
}

# AWS Config Rule that checks whether logging is enabled for your S3 buckets.
resource "aws_config_organization_managed_rule" "s3_bucket_logging_organization_config_rules" {
  name              = "s3-bucket-logging-enabled"
  rule_identifier   = "S3_BUCKET_LOGGING_ENABLED"
}

# AWS Config Rule that checks whether logging is enabled for your S3 buckets.
resource "aws_config_organization_managed_rule" "s3_bucket_encryption_organization_config_rules" {
  name              = "s3-bucket-server-side-encryption-enabled"
  rule_identifier   = "S3_BUCKET_SERVER_SIDE_ENCRYPTION_ENABLED"
}

terraform適用

あとはいつもの通り以下でリソース作成

terraform init
terraform plan
terraform apply

動作確認

集約したアカウントにログイン

Configのアグリゲータにアクセスしたら
全てのアカウントの非準拠ルールとか出てる!きっとうまく行ってそう!

詰まったところ

別アカウントからのS3アクセスがうまくいかない

terraform applyするとconfig channel resourceあたりでこんなエラー出てだいぶ詰まった

nsufficient delivery policy to s3 bucket:<Bucket Name>, unable to write to bucket, provided s3 key prefix is 'null'.

ここの説明にある通り、集約するS3 bucketのpolicyとIAM RoleのPolicyをちゃんと合わせないとここから抜け出せない

https://aws.amazon.com/jp/premiumsupport/knowledge-center/config-console-error/

moduleでfor_each, countうまくいかない

この方の記事にもあるとおり

https://kazuhira-r.hatenablog.com/entry/2020/07/04/161848

カッコつけてmoduleで分離して色々variablesで渡してたら
こんなエラーメッセージが出て、なかなか抜け出せなかった・・・

The "for_each" value depends on resource attributes that cannot be determined
until apply, so Terraform cannot predict how many instances will be created.
To work around this, use the -target argument to first apply only the
resources that the for_each depends on.

ちょっと戦ったけど、module使わない方で逃げた🙈

まとめ

だいぶ時間かかった・・・
Configで遊び始めたらついに個人アカウントにも50円ほど課金が発生し始めた・・・

最近転職してお給金も減ったので
一旦このConfigのrecordを止めておこう。これで請求一旦止まるかな・・?

ここ変更してterraform applyすると記録止まりまっす

 resource "aws_config_configuration_recorder_status" "config_recorder_status" {
   name       = aws_config_configuration_recorder.config_recorder.name
-  is_enabled = true
+  is_enabled = false
   depends_on = [aws_config_delivery_channel.config_channel]
 }

Discussion

ログインするとコメントできます