Managing Multiple Terraform Backends in a Single S3 Bucket

Introduction

When building infrastructure with Terraform, you likely encounter situations where you need to manage multiple State files for each project or environment. On the other hand, creating an S3 bucket for every State file can lead to an explosion of buckets, making management cumbersome.
In this article, I will introduce a method to manage multiple Terraform Backends using a single S3 bucket.

Why are multiple backends necessary?

Terraform manages the state of infrastructure by sharing state files, but backends are often separated for the following reasons:

• To separate environments such as Development / Staging / Production
• To manage Terraform configurations separately for each application

Since each configuration requires its own backend definition, management becomes difficult if you create a new S3 bucket every time.

Consolidating into a single S3 bucket by organizing keys

By specifying a hierarchical path in the key of Terraform's backend "s3", you can store multiple state files in one bucket.

app1/dev/main.tf

terraform {
  backend "s3" {
    bucket       = "my-terraform-backend"
    key          = "app1/dev/terraform.tfstate"
    region       = "ap-northeast-1"
    use_lockfile = true
  }
}

For other configurations, you're good to go by just changing the path of the key in the same bucket like this 👍

app2/stg/main.tf

terraform {
  backend "s3" {
    bucket       = "my-terraform-backend"
    key          = "app2/stg/terraform.tfstate"
    region       = "ap-northeast-1"
    use_lockfile = true
  }
}

S3 Native State Locking, which became GA in Terraform 1.11, also works perfectly 🎉

If you are using DynamoDB locking, it is also fine to specify a common DynamoDB table in the dynamodb_table of the backend "s3".

Conclusion

With such a simple configuration, you can manage multiple States!
I hope this helps your Terraform life 🎏