Securely Setting Aurora (RDS) Passwords in Terraform

Hello.

Recently, I have been using Terraform at work to build web systems using AWS Aurora.

When setting up Aurora, you need to specify the master password and username for the cluster.

It looks something like this:

resource "aws_rds_cluster" "main" {
  cluster_identifier              = var.cluster_settings.cluster_name
  engine                          = var.cluster_settings.engine
  engine_version                  = var.cluster_settings.engine_version
  availability_zones              = var.cluster_settings.availability_zones
  database_name                   = var.cluster_settings.database_name
  # Username and password
  master_username                 = var.cluster_settings.master_username
  master_password                 = var.cluster_settings.master_password
  # Omitted
}

However, if you pass the password within the *.tf file here, the password information will remain in the tfstate file.

Reference articles:

• https://techlife.cookpad.com/entry/2020/02/28/120000
• Terraform Official (Purpose of State Files)
  • https://www.terraform.io/language/state/purpose

When searching for ways to handle sensitive information in Terraform, an article from the "Mobarab Engineer Blog" lists several methods.

https://engineering.mobalab.net/2021/03/25/handling-secrets-with-terraform/

Among them, I felt that "Method 4: Do not handle sensitive information in Terraform" is ultimately the most secure, so I decided to put it into practice this time.

When you specify sensitive information such as passwords while creating resources with Terraform, it is inevitable that it will be written to the state file, even if it is retrieved from an external source like environment variables or Secrets Manager. While managing it securely using remote state is generally sufficient, depending on security requirements, Methods 1 to 3 might not be acceptable.

Method 4 basically involves the following:

• Keep sensitive information in Secrets Manager, etc.
• Specify a placeholder/dummy value for sensitive information when creating resources with Terraform
• Change the sensitive information via another method after resource creation
  It takes a bit more effort, but by using Terraform's null_resource or local-exec, you can trigger a separate script after resource creation is complete to fetch the value from Secrets Manager and set it.

Implementation Details

https://github.com/bun913/aws_network_practice/tree/main/aurora

The full code shown below is located in the corresponding directory of the repository above.

What I did this time was:

Preparation

• Ensure the IAM user has sufficient permissions
  • Network, EC2, SSM, Aurora creation, Aurora password changes, etc.
• Store the information for the RDS master password in the SSM Parameter Store in advance
  • This part is done manually, not through Terraform
  • Note: While I used Parameter Store this time, Secrets Manager is better if you have requirements for periodic password rotation
  • Reference: https://qiita.com/tomoya_oka/items/a3dd44879eea0d1e3ef5

Resources built with Terraform

• EC2 instance and network resources for connection verification to Aurora
  • The EC2 instance is placed in a private subnet. Connect securely using SSM Session Manager
  • Reference: https://dev.classmethod.jp/articles/choosing-the-right-shell-access-solution-to-aws-ec2/
• Network for Aurora multi-AZ configuration and creation of Aurora instances

Creating the Management EC2

I will omit the creation of network resources and security groups, but I will create an EC2 instance.

resource "aws_instance" "bastion" {
  ami                    = data.aws_ssm_parameter.amzn2_ami.value
  instance_type          = "t2.micro"
  iam_instance_profile   = aws_iam_instance_profile.bastion-profile.name
  subnet_id              = var.bastion_subnet
  vpc_security_group_ids = [aws_security_group.allow_https_outbound.id]
  user_data              = file("${path.module}/user_data.sh")
  tags                   = var.tags
}

To ensure the latest version of amazon-ssm-agent can be retrieved, the user_data is set to the following content:

#!/bin/bash
cd /tmp
sudo yum install -y https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/linux_amd64/amazon-ssm-agent.rpm
sudo systemctl start amazon-ssm-agent

For more details, please refer to the following directory:

https://github.com/bun913/aws_network_practice/tree/main/aurora/modules/bastion

In this setup, the EC2 instance is placed in a private subnet and does not utilize a NAT gateway. Since Amazon Linux 2 is used, I created a VPC endpoint for S3 so that ssm-agent can be updated to the latest version.

Reference: https://aws.amazon.com/jp/premiumsupport/knowledge-center/ec2-al1-al2-update-yum-without-internet/

Additionally, several VPC endpoints need to be created to use SSM, so I have created those as well.

References:

• https://aws.amazon.com/jp/premiumsupport/knowledge-center/systems-manager-ec2-instance-not-appear/
• https://dev.classmethod.jp/articles/troubleshooting-ssm-session-manager-configuration/

As a side note, Classmethod's articles are really divinely easy to understand.

Creating the Aurora Cluster

Create the Aurora cluster and DB instances as follows. Other resources like parameter groups are also created (https://github.com/bun913/aws_network_practice/tree/main/aurora/modules/db).

# rds cluster group
resource "aws_rds_cluster" "main" {
  cluster_identifier              = var.cluster_settings.cluster_name
  engine                          = var.cluster_settings.engine
  engine_version                  = var.cluster_settings.engine_version
  availability_zones              = var.cluster_settings.availability_zones
  database_name                   = var.cluster_settings.database_name
  master_username                 = var.cluster_settings.master_username
  master_password                 = var.cluster_settings.master_password
  backup_retention_period         = var.cluster_settings.backup_retention_period
  db_subnet_group_name            = aws_db_subnet_group.main.id
  vpc_security_group_ids          = [aws_security_group.db.id]
  db_cluster_parameter_group_name = aws_rds_cluster_parameter_group.main.id
  # NOTE: Deletion protection is set to false for testing. Be sure to enable it in production.
  deletion_protection = false
  skip_final_snapshot = true
}

# Change the cluster password
resource "null_resource" "cluster" {
  # Triggered when the DB is created
  triggers = {
    cluster_instance_ids = aws_rds_cluster.main.id
  }
  provisioner "local-exec" {
    # Change the RDS password using the AWS CLI
    command    = "aws ssm get-parameter --name /${var.project}/db_password --query 'Parameter.Value' --with-decryption | xargs -IPASS aws rds modify-db-cluster --db-cluster-identifier ${aws_rds_cluster.main.id} --master-user-password PASS" --apply-immediately
    on_failure = fail
  }
}

# rds_instance
resource "aws_rds_cluster_instance" "example" {
  count                   = 2
  cluster_identifier      = aws_rds_cluster.main.id
  identifier              = "${aws_rds_cluster.main.id}-${count.index}"
  engine                  = aws_rds_cluster.main.engine
  engine_version          = aws_rds_cluster.main.engine_version
  instance_class          = var.instance_class
  db_subnet_group_name    = aws_db_subnet_group.main.name
  db_parameter_group_name = aws_db_parameter_group.main.name
  publicly_accessible = false
}

The key point is the following section:

# Change the cluster password
resource "null_resource" "cluster" {
  # Triggered when the DB is created
  triggers = {
    cluster_instance_ids = aws_rds_cluster.main.id
  }
  provisioner "local-exec" {
    # Change the RDS password using the AWS CLI
    command    = "aws ssm get-parameter --name /${var.project}/db_password --query 'Parameter.Value' --with-decryption | xargs -IPASS aws rds modify-db-cluster --db-cluster-identifier ${aws_rds_cluster.main.id} --master-user-password PASS"
    on_failure = fail
  }
}

I am using null_resource to trigger the RDS password change via local-exec once the DB is created.

By doing this, the password retrieval and update are performed outside of Terraform, so the actual password is not left in either the code or the tfstate file (only the dummy password set initially remains).

Connection Verification

After completing the steps above, launch Session Manager from the AWS Management Console and connect from the EC2 instance using the real password...

Connection confirmed successfully! (The endpoint for the Aurora Writer instance is specified in the red-boxed area.)

Conclusion

As I went through this process, I came to the conclusion that avoiding the direct handling of sensitive information in Terraform is likely the safest approach.

I've been figuring this out as I go, so please let me know if you have any other better methods!