iTranslated by AI
ecspresso advent calendar 2020 day 19 - Terraform tfstate
This is day 19 of the ecspresso Advent calendar, where we are summarizing how to use ecspresso, a deployment tool for Amazon ECS.
External Resources Described in Definition Files
Service and task definition files used by ecspresso may require IDs or ARNs (Amazon Resource Names) of related external resources.
In the following example, the ARN of a Load Balancer target group and the IDs for a VPC security group and subnets are described in the service definition file.
{
"launchType": "FARGATE",
"loadBalancers": [
{
"containerName": "nginx",
"containerPort": 80,
"targetGroupArn": "arn:aws:elasticloadbalancing:ap-northeast-1:12345789012:targetgroup/EC2Co-Defau-SA0YF35VBLCJ/b491afabf44765de"
}
],
"networkConfiguration": {
"awsvpcConfiguration": {
"assignPublicIp": "ENABLED",
"securityGroups": [
"sg-043eab2d606362f03"
],
"subnets": [
"subnet-0089ed3e1bdff1fc9",
"subnet-0d750adbd139f411d"
]
}
}
}
Since ecspresso does not manage these external resources, it cannot automatically know their IDs. However, hardcoding ID or ARN values directly into definition files causes issues with maintainability and readability.
Loading Terraform State Files
Terraform is an open-source tool developed by HashiCorp for managing cloud infrastructure as code. As an IaC (Infrastructure as Code) tool for AWS, it stands alongside the official CloudFormation as a primary choice.
ecspresso can read Terraform State files (tfstate) and refer to attributes such as IDs or ARNs using the resource names defined therein.
To use this feature, describe the path to the terraform.tfstate file in your configuration file. The path is relative to the location of the configuration file.
plugins:
- name: tfstate
config:
path: terraform.tfstate
Assume that the following Terraform .tf file has been prepared to manage the necessary resources:
provider "aws" {
region = "ap-northeast-1"
}
resource "aws_vpc" "ecspresso-demo" {
cidr_block = "10.0.0.0/16"
}
resource "aws_subnet" "az-a" {
cidr_block = "10.0.0.0/24"
vpc_id = aws_vpc.ecspresso-demo.id
}
resource "aws_subnet" "az-c" {
cidr_block = "10.0.1.0/24"
vpc_id = aws_vpc.ecspresso-demo.id
}
data "aws_security_group" "http" {
id = "sg-043eab2d606362f03"
}
resource "aws_lb_target_group" "ecspresso-demo" {
name = "EC2Co-Defau-SA0YF35VBLCJ"
port = 80
protocol = "HTTP"
target_type = "ip"
vpc_id = aws_vpc.ecspresso-demo.id
}
In this example, only the security group is defined as a Data source.
Referencing tfstate in Definition Files
By using the template function tfstate in the definition file, you can reference resources within the tfstate file using Terraform's resource naming format. Rewriting the service definition file shown earlier to use tfstate references looks like this:
{
"launchType": "FARGATE",
"loadBalancers": [
{
"containerName": "nginx",
"containerPort": 80,
"targetGroupArn": "{{ tfstate `aws_lb_target_group.ecspresso-demo.arn` }}"
}
],
"networkConfiguration": {
"awsvpcConfiguration": {
"assignPublicIp": "ENABLED",
"securityGroups": [
"{{ tfstate `data.aws_security_group.http.id` }}"
],
"subnets": [
"{{ tfstate `aws_subnet.az-a.id` }}",
"{{ tfstate `aws_subnet.az-c.id` }}"
]
}
}
}
Any resource attribute present in the tfstate, besides ID or ARN, can be referenced and expanded in the template.
Now, raw IDs and ARNs have been removed from the definition files, and everything can be resolved by name.
You can actually render the files using ecspresso render --service-definition, or use ecspresso diff to check for differences against the state of the service on ECS.
Advantages of tfstate Referencing
Compared to directly describing IDs, referencing tfstate offers several advantages:
Improved Readability
Descriptions become much easier for humans to understand than random strings of IDs.
If you describe a name that does not exist in the tfstate, the template expansion will fail, making it easy to discover typos before deployment.
No Need to Modify Definition Files Across Environments
This is particularly useful when different physical resources exist for staging and production environments.
As long as the different tfstates for each environment manage resources with the same names, you can use the same definition files simply by changing the tfstate that ecspresso references to resolve to the correct environment-specific IDs.
tfstate Referencing Tips
Handling Resource Names in Array Format
When Terraform resources are defined using for_each or similar methods, the resource address might use array reference syntax like aws_subnet.az["a"]. If you pass this directly to the tfstate function as shown below, it will break the JSON structure:
"{{ tfstate `aws_subnet.az["a"].id` }}"
As explained on Day 5: Template Syntax, although ecspresso can load the file even if the structure is broken, it causes issues when editing with an editor.
Therefore, although it is not standard Terraform syntax, for convenience, ecspresso allows single quotes to be used interchangeably with double quotes in array-style descriptions.
"{{ tfstate `aws_subnet.az['a'].id` }}"
How to Check Values Actually Present in tfstate
Since tfstate is in JSON format, humans can read it directly. However, it is difficult to find a resource by searching for its Terraform address as a string. Here are two convenient methods for verification.
Using terraform state list / show
You can use the list and show subcommands of the terraform state command to check elements within a tfstate file.
$ terraform state list -state=terraform.tfstate
data.aws_security_group.http
aws_lb_target_group.ecspresso-demo
aws_subnet.az-a
aws_subnet.az-c
aws_vpc.ecspresso-demo
$ terraform state show -state=terraform.tfstate aws_subnet.az-a
# aws_subnet.az-a:
resource "aws_subnet" "az-a" {
arn = "arn:aws:ec2:ap-northeast-1:123456789012:subnet/subnet-0d750adbd139f411d"
assign_ipv6_address_on_creation = false
availability_zone = "ap-northeast-1a"
availability_zone_id = "apne1-az4"
cidr_block = "10.0.0.0/24"
id = "subnet-0d750adbd139f411d"
map_public_ip_on_launch = false
owner_id = "314472643515"
tags = {}
vpc_id = "vpc-0f24baede5950eb2d"
timeouts {}
}
Using tfstate-lookup
The package used by ecspresso to implement tfstate referencing is github.com/fujiwara/tfstate-lookup/tfstate.
By using the tfstate-lookup command, you can list and reference resources within a tfstate file.
Since ecspresso actually uses tfstate-lookup/tfstate rather than terraform state, using this tool ensures you are seeing exactly what will be expanded in your templates. Additionally, it is much faster as it is specialized solely for reading tfstate.
$ tfstate-lookup -s terraform.tfstate
data.aws_security_group.http
aws_lb_target_group.ecspresso-demo
aws_subnet.az-a
aws_subnet.az-c
aws_vpc.ecspresso-demo
$ tfstate-lookup -s terraform.tfstate aws_subnet.az-a
{
"arn": "arn:aws:ec2:ap-northeast-1:123456789012:subnet/subnet-0d750adbd139f411d",
"assign_ipv6_address_on_creation": false,
"availability_zone": "ap-northeast-1a",
"availability_zone_id": "apne1-az4",
"cidr_block": "10.0.0.0/24",
"id": "subnet-0d750adbd139f411d",
"ipv6_cidr_block": "",
"ipv6_cidr_block_association_id": "",
"map_public_ip_on_launch": false,
"outpost_arn": "",
"owner_id": "123456789012",
"tags": {},
"timeouts": {
"create": null,
"delete": null
},
"vpc_id": "vpc-0f24baede5950eb2d"
}
Performance comparison (Terraform v0.14.2, tfstate-lookup v0.0.14):
$ time terraform state list -state=terraform.tfstate > /dev/null
real 0m0.811s
user 0m0.742s
sys 0m0.158s
$ time tfstate-lookup -s terraform.tfstate > /dev/null
real 0m0.019s
user 0m0.018s
sys 0m0.010s
On Day 20, we will explain how to generate task and service definitions using the template language Jsonnet.
Discussion