🦔

委任したAccountでAWS Configを観測 by Terraform

2021/05/09に公開

概要

またAWS Landing Zone環境構築の続き
今回はTerraformを用いてマルチアカウント環境でAWS Configを設定する

ポイントとしては以下

  1. Log Archive Accountに組織Aggregatorを委任
  2. 全アカウントのConfig計測データをLog Archive AccountのS3に集約
    1. 各アカウントのConfig設定を一つのterraformで実施
  3. 組織内全アカウントにConfig Ruleを適用

イメージとしては以下のような感じ

ディレクトリ構成

今回こんな感じで作成した
(やはりTerraform歴浅いのでまだまだ実力不足感を感じる

$ tree
.
├── backend.tf
├── main.tf
├── modules
│   └── config
│       ├── aggregator
│       │   ├── main.tf
│       │   ├── outputs.tf
│       │   └── variables.tf
│       ├── each-config
│       │   ├── main.tf
│       │   ├── outputs.tf
│       │   └── variables.tf
│       └── rules
│           ├── main.tf
│           ├── outputs.tf
│           └── variables.tf
├── provider.tf
└── versions.tf

5 directories, 13 files

やったこと

それぞれやったことのポイントをメモしておく

各アカウント情報の設定

各AWSアカウントにConfig設定するために
providerのassume_roleにrole_arnを指定する

main.tfでこのproviderのaliasを指定することで
対象のAWSアカウントに対してリソースを作成したりできる🤓

roleは前回のSSO記事で生成されたAdministratorAccessの付与されたrole使ってる

./provider.tf
provider "aws" {
  region  = "ap-northeast-1"
}

provider "aws" {
  alias = "log-archive-production"

  region  = "ap-northeast-1"
  assume_role {
    role_arn = "arn:aws:iam::${aws_organizations_account.log_archive_production_account.id}:role/OrganizationAccountAccessRole"
  }
}

provider "aws" {
  alias = "ucwork-production-account"

  region  = "ap-northeast-1"
  assume_role {
    role_arn = "arn:aws:iam::${aws_organizations_account.ucwork_production_account.id}:role/OrganizationAccountAccessRole"
  }
}

# ...アカウントの分だけproviderを用意する

各リソース作成指示

各種moduleやresourceの作成
Organizational Unit(OU), Account, SSOは
コメントに記載した別記事参照くだされ

./main.tf
# -----------------------------------------------------------
# Set up organizations
# -----------------------------------------------------------

// この辺参照ください
https://zenn.dev/ucwork/articles/abebfe4003a7ee

// OUとAccountを管理してる場所

# -----------------------------------------------------------
# Set up aggregator and each config in all account
# -----------------------------------------------------------
// terraformのresourceないみたいなのでawsコマンドで実行
// 登録済みの場合エラーになるのでon_failure = continueをセット
resource "null_resource" "config_delegated" {
  provisioner "local-exec" {
    command = "aws organizations register-delegated-administrator --account-id ${aws_organizations_account.log_archive_production_account.id} --service-principal config.amazonaws.com"
    on_failure = continue
  }
}

resource "null_resource" "config_multi_setup_delegated" {
  provisioner "local-exec" {
    command = "aws organizations register-delegated-administrator --account-id ${aws_organizations_account.log_archive_production_account.id} --service-principal config-multiaccountsetup.amazonaws.com"
    on_failure = continue
  }
  depends_on = [ null_resource.config_delegated ]
}

module "config_aggregator" {
  source = "./modules/config/aggregator"

  aggregator_account_id = aws_organizations_account.log_archive_production_account.id
  aggregator_s3_region = "ap-northeast-1"

  providers = {
    aws = aws.log-archive-production
  }
}

// config_aggregatorで作成したS3使いたいので依存関係(depends_on)指定
module "config_management" {
  source = "./modules/config/each-config"

  bucket_arn = module.config_aggregator.config_s3_arn
  bucket_id = module.config_aggregator.config_s3_id

  depends_on = [
    module.config_aggregator
  ]
}

// provider.tfで指定したaliasで各アカウントにresource作成
module "config_ucwork_production" {
  source = "./modules/config/each-config"

  bucket_arn = module.config_aggregator.config_s3_arn
  bucket_id = module.config_aggregator.config_s3_id

  providers = {
    aws = aws.ucwork-production-account
  }
  depends_on = [
    module.config_aggregator
  ]
}

# Config設定するアカウントの分だけ上記each-configのmoduleを追記する!
// moduleのfor_each使いたかったけど、providersのaliasにeach.value指定するとエラーで怒られたので愚直に全アカウント分書いた・・・

# -----------------------------------------------------------
# Set up organization config rules
# -----------------------------------------------------------
// organizationに紐づくアカウント全部にルール作成
// 各アカウントのconfig resourceないと怒られるのでdepends_onを指定
module "config_rules" {
  source = "./modules/config/rules"

  depends_on = [
    module.config_management,
    module.config_shared_services_production,
    module.config_ucwork_production,
    module.config_ucwork_sdlc
  ]
}

# -----------------------------------------------------------
# Set up Single Sign-On (SSO)
# -----------------------------------------------------------

// この辺見てね
https://zenn.dev/ucwork/articles/7f30488b6d2bbf

// SSOでマルチアカウントアクセスするための設定まわりが書いてある

Aggregatorアカウントの設定

./modules/config/aggregator/main.tf
# logging s3
resource "aws_s3_bucket" "config_bucket" {
  bucket = "config-bucket-${var.aggregator_account_id}-${var.aggregator_s3_region}"
  acl    = "private"

  server_side_encryption_configuration {
    rule {
      apply_server_side_encryption_by_default {
        sse_algorithm = "AES256"
      }
    }
  }
}

resource "aws_s3_bucket_policy" "config_logging_policy" {
  bucket = aws_s3_bucket.config_bucket.id
  policy = <<POLICY
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AWSConfigBucketPermissionsCheck",
      "Effect": "Allow",
      "Principal": {
        "Service": [
         "config.amazonaws.com"
        ]
      },
      "Action": "s3:GetBucketAcl",
      "Resource": "${aws_s3_bucket.config_bucket.arn}"
    },
    {
      "Sid": "AWSConfigBucketExistenceCheck",
      "Effect": "Allow",
      "Principal": {
        "Service": [
          "config.amazonaws.com"
        ]
      },
      "Action": "s3:ListBucket",
      "Resource": "${aws_s3_bucket.config_bucket.arn}"
    },
    {
      "Sid": "AWSConfigBucketDelivery",
      "Effect": "Allow",
      "Principal": {
        "Service": [
         "config.amazonaws.com"
        ]
      },
      "Action": "s3:PutObject",
      "Resource": "${aws_s3_bucket.config_bucket.arn}/AWSLogs/*/Config/*",
      "Condition": {
        "StringEquals": {
          "s3:x-amz-acl": "bucket-owner-full-control"
        }
      }
    }
  ]
}
POLICY
}

# iam role
resource "aws_iam_role" "config_role" {
  name = "config_aggregator_role"

  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": "config.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF
}

resource "aws_iam_policy" "config_org_policy" {
  path        = "/"
  description = "IAM Policy for AWS Config"
  name        = "ConfigPolicy"

  policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "config:GetOrganizationConfigRuleDetailedStatus",
        "config:Put*",
        "iam:GetPasswordPolicy",
        "organizations:ListAccounts",
        "organizations:DescribeOrganization",
        "organizations:ListAWSServiceAccessForOrganization",
        "organization:EnableAWSServiceAccess"
      ],
      "Effect": "Allow",
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": ["s3:PutObject"],
      "Resource": ["${aws_s3_bucket.config_bucket.arn}/AWSLogs/${var.aggregator_account_id}/*"],
      "Condition": {
        "StringLike": {
          "s3:x-amz-acl": "bucket-owner-full-control"
        }
      }
    },
    {
      "Effect": "Allow",
      "Action": ["s3:GetBucketAcl"],
      "Resource": "${aws_s3_bucket.config_bucket.arn}"
    }
  ]
}
EOF
}

resource "aws_iam_role_policy_attachment" "config_org_policy_attach" {
  role       = aws_iam_role.config_role.name
  policy_arn = aws_iam_policy.config_org_policy.arn
}

resource "aws_iam_role_policy_attachment" "config_policy_attach" {
  role       = aws_iam_role.config_role.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSConfigRole"
}

resource "aws_iam_role_policy_attachment" "read_only_policy_attach" {
  role       = aws_iam_role.config_role.name
  policy_arn = "arn:aws:iam::aws:policy/ReadOnlyAccess"
}

# -----------------------------------------------------------
# set up the aggregator account Config
# -----------------------------------------------------------
resource "aws_config_configuration_aggregator" "organization" {
  name = "organization-aggregator"

  organization_aggregation_source {
    all_regions = true
    role_arn    = aws_iam_role.config_role.arn
  }
}

resource "aws_config_configuration_recorder" "config_recorder" {
  role_arn = aws_iam_role.config_role.arn

  recording_group {
    all_supported                 = true
    include_global_resource_types = true
  }
}

# Delivery channel resource and bucket location to specify configuration history location.
resource "aws_config_delivery_channel" "config_channel" {
  s3_bucket_name = aws_s3_bucket.config_bucket.id
  depends_on = [aws_config_configuration_recorder.config_recorder]
}

resource "aws_config_configuration_recorder_status" "config_recorder_status" {
  name       = aws_config_configuration_recorder.config_recorder.name
  is_enabled = true
  depends_on = [aws_config_delivery_channel.config_channel]
}
./modules/config/aggregator/variables.tf
variable "aggregator_account_id" {
  type        = string
  default     = ""
  description = "config aggregator account id"
}

variable "aggregator_s3_region" {
  type        = string
  default     = ""
  description = "config aggregator s3 region"
}
./modules/config/aggregator/outputs.tf
output "config_s3_arn" {
    value = aws_s3_bucket.config_bucket.arn
    description = "AWS Config Aggregator s3 bucket arn"
}

output "config_s3_id" {
    value = aws_s3_bucket.config_bucket.id
    description = "AWS Config Aggregator s3 bucket name(id)"
}

各アカウントのConfig設定

./modules/config/each-config/main.tf
resource "aws_iam_role" "config_role" {
  name = "ConfigRecorderRole"

  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": "config.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF
}

resource "aws_iam_policy" "s3_config_org_policy" {
  path        = "/"
  description = "S3ConfigOrganizationPolicy"

  policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:PutObject"],
      "Resource": ["${var.bucket_arn}/AWSLogs/*/Config/*"],
      "Condition": {
        "StringLike": {
          "s3:x-amz-acl": "bucket-owner-full-control"
        }
      }
    },
    {
      "Effect": "Allow",
      "Action": ["s3:GetBucketAcl"],
      "Resource": "${var.bucket_arn}"
    }
  ]
}
EOF
}

resource "aws_iam_role_policy_attachment" "config_s3_policy_attach" {
  role       = aws_iam_role.config_role.name
  policy_arn = aws_iam_policy.s3_config_org_policy.arn
}

resource "aws_iam_role_policy_attachment" "read_only_attachment" {
  role       = aws_iam_role.config_role.name
  policy_arn = "arn:aws:iam::aws:policy/ReadOnlyAccess"
}

resource "aws_iam_role_policy_attachment" "config_attachment" {
  role       = aws_iam_role.config_role.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSConfigRole"
}

# -----------------------------------------------------------
# set up the Config
# -----------------------------------------------------------
resource "aws_config_configuration_recorder" "config_recorder" {
  role_arn = aws_iam_role.config_role.arn
}

resource "aws_config_delivery_channel" "config_channel" {
  s3_bucket_name = var.bucket_id
  depends_on = [aws_config_configuration_recorder.config_recorder]
}

resource "aws_config_configuration_recorder_status" "config_recorder_status" {
  name       = aws_config_configuration_recorder.config_recorder.name
  is_enabled = true
  depends_on = [aws_config_delivery_channel.config_channel]
}
./modules/config/each-config/variables.tf
variable "bucket_arn" {
  type        = string
  default     = ""
  description = "organization config s3 bucket arn"
}

variable "bucket_id" {
  type        = string
  default     = ""
  description = "organization config s3 bucket name(id)"
}

config ruleの設定

./modules/config/rules/main.tf
# AWS Config Rule that manages IAM Password Policy
resource "aws_config_organization_managed_rule" "iam_policy_organization_config_rule" {
  input_parameters  = <<EOF
    {
      "RequireUppercaseCharacters": "true",
      "RequireLowercaseCharacters": "true",
      "RequireSymbols": "true",
      "RequireNumbers": "true",
      "MinimumPasswordLength": "9",
      "PasswordReusePrevention": "5",
      "MaxPasswordAge": "90"
    }
  EOF

  name              = "iam-password-policy"
  rule_identifier   = "IAM_PASSWORD_POLICY"
}

# AWS Config Rule that manages IAM Root Access Keys to see if they exist
resource "aws_config_organization_managed_rule" "iam_root_access_key_organization_config_rule" {
  name              = "iam-root-access-key-check"
  rule_identifier   = "IAM_ROOT_ACCESS_KEY_CHECK"
}

# AWS Config Rule that checks whether your AWS account is enabled to use multi-factor authentication (MFA)
# hardware device to sign in with root credentials.
resource "aws_config_organization_managed_rule" "root_hardware_mfa_organization_config_rule" {
  name              = "root-hardware-mfa"
  rule_identifier   = "ROOT_ACCOUNT_HARDWARE_MFA_ENABLED"
}

# AWS Config Rule that checks whether users of your AWS account require a multi-factor authentication (MFA)
# device to sign in with root credentials.
resource "aws_config_organization_managed_rule" "root_account_mfa_organization_config_rules" {
  name              = "root-account-mfa-enabled"
  rule_identifier   = "ROOT_ACCOUNT_MFA_ENABLED"
}


# AWS Config Rule that checks whether the required public access block settings are configured from account level.
# The rule is only NON_COMPLIANT when the fields set below do not match the corresponding fields in the configuration
# item.
resource "aws_config_organization_managed_rule" "s3_public_access_organization_config_rules" {
  name              = "s3-account-level-public-access-blocks"
  rule_identifier   = "S3_ACCOUNT_LEVEL_PUBLIC_ACCESS_BLOCKS"
}

# AWS Config Rule that checks whether logging is enabled for your S3 buckets.
resource "aws_config_organization_managed_rule" "s3_bucket_logging_organization_config_rules" {
  name              = "s3-bucket-logging-enabled"
  rule_identifier   = "S3_BUCKET_LOGGING_ENABLED"
}

# AWS Config Rule that checks whether logging is enabled for your S3 buckets.
resource "aws_config_organization_managed_rule" "s3_bucket_encryption_organization_config_rules" {
  name              = "s3-bucket-server-side-encryption-enabled"
  rule_identifier   = "S3_BUCKET_SERVER_SIDE_ENCRYPTION_ENABLED"
}

terraform適用

あとはいつもの通り以下でリソース作成

terraform init
terraform plan
terraform apply

動作確認

集約したアカウントにログイン

Configのアグリゲータにアクセスしたら
全てのアカウントの非準拠ルールとか出てる!きっとうまく行ってそう!

詰まったところ

別アカウントからのS3アクセスがうまくいかない

terraform applyするとconfig channel resourceあたりでこんなエラー出てだいぶ詰まった

nsufficient delivery policy to s3 bucket:<Bucket Name>, unable to write to bucket, provided s3 key prefix is 'null'.

ここの説明にある通り、集約するS3 bucketのpolicyとIAM RoleのPolicyをちゃんと合わせないとここから抜け出せない
https://aws.amazon.com/jp/premiumsupport/knowledge-center/config-console-error/

moduleでfor_each, countうまくいかない

この方の記事にもあるとおり
https://kazuhira-r.hatenablog.com/entry/2020/07/04/161848

カッコつけてmoduleで分離して色々variablesで渡してたら
こんなエラーメッセージが出て、なかなか抜け出せなかった・・・

The "for_each" value depends on resource attributes that cannot be determined
until apply, so Terraform cannot predict how many instances will be created.
To work around this, use the -target argument to first apply only the
resources that the for_each depends on.

ちょっと戦ったけど、module使わない方で逃げた🙈

まとめ

だいぶ時間かかった・・・
Configで遊び始めたらついに個人アカウントにも50円ほど課金が発生し始めた・・・

最近転職してお給金も減ったので
一旦このConfigのrecordを止めておこう。これで請求一旦止まるかな・・?

ここ変更してterraform applyすると記録止まりまっす

 resource "aws_config_configuration_recorder_status" "config_recorder_status" {
   name       = aws_config_configuration_recorder.config_recorder.name
-  is_enabled = true
+  is_enabled = false
   depends_on = [aws_config_delivery_channel.config_channel]
 }

Discussion