Automating AWS SAM Application Deployment with CircleCI Orbs

Hello, I'm Tomada.

As a member of the CircleCI Blogger Community, I'm writing an article about using CircleCI Orbs to automatically deploy an AWS SAM application.

Up until now, I had rarely used CircleCI Orbs, but I decided to give them a try after hearing they allow for much more concise configurations than writing out every single setting manually.
This time, I happened to notice the existence of an Orb specifically for AWS SAM, which allows you to deploy serverless applications as code. I'll be using aws-sam-serverless@3.0.0 while referring to the official documentation.

Note that I have introduced the method of deploying using only the AWS SAM CLI in a separate article.
In that article, I also cover running tests locally using Docker, so I recommend checking it out at least once.
→Deploying a Hello World Application Using AWS SAM

What are CircleCI Orbs?

I'll quote from the official site.

CircleCI Orbs are shareable open-source packages of reusable configuration elements, including jobs, commands, and executors. Orbs allow you to simplify your configuration and quickly and easily integrate with software and service stacks across many projects.

To put it very simply, I think of them as pre-prepared, convenient environment packages.
In fact, for the Orb I used this time, the CircleCI configuration file, which would have been hundreds of lines without Orbs, was reduced to about 30 lines by leveraging Orbs.
It's incredibly powerful.

Prerequisites

• Create an AWS Account
  • Create an IAM user with FullAccess permissions for Lambda, API Gateway, S3, and IAM.
  • (In a real environment, you should restrict access permissions to the minimum necessary.)
  • Keep note of the AWS Access Key and AWS Secret Key.
• Install AWS SAM CLI locally
  • It's okay if running "sam --version" displays the version.
• Create a CircleCI account
  • Sign in using GitHub.
• Create a GitHub repository
  • Create a repository where you can push the application you'll build this time.

Building the SAM Application

Downloading the Sample Application

We'll download an application that can be used as a sample through the sam command.
First, enter the following command.

$ sam init

Along the way, you'll choose the template source, package type, and runtime.
The project name can be anything you like.
Here, I'm using "hello-sam".

$ sam init
Which template source would you like to use?
        1 - AWS Quick Start Templates
        2 - Custom Template Location
Choice: 1
What package type would you like to use?
        1 - Zip (artifact is a zip uploaded to S3)
        2 - Image (artifact is an image uploaded to an ECR image repository)
Package type: 1

Which runtime would you like to use?
        1 - nodejs14.x
        2 - python3.8
        3 - ruby2.7
        4 - go1.x
        5 - java11
        6 - dotnetcore3.1
        7 - nodejs12.x
        8 - nodejs10.x
        9 - python3.7
        10 - python3.6
        11 - python2.7
        12 - ruby2.5
        13 - java8.al2
        14 - java8
        15 - dotnetcore2.1
Runtime: 9

Project name [sam-app]: hello-sam

Once you've entered these, the template will be automatically downloaded from GitHub to your local machine.
Finally, select the template.

Cloning app templates from https://github.com/aws/aws-sam-cli-app-templates

AWS quick start application templates:
        1 - Hello World Example
        2 - EventBridge Hello World
        3 - EventBridge App from scratch (100+ Event Schemas)
        4 - Step Functions Sample App (Stock Trader)
Template selection: 1

    -----------------------
    Generating application:
    -----------------------
    Name: hello-sam
    Runtime: python3.7
    Dependency Manager: pip
    Application Template: hello-world
    Output Directory: .

    Next steps can be found in the README file at ./hello-sam/README.md

If you see this, the initial preparation is complete.

Building the Application

Let's move to the project directory and check its contents.

$ cd hello-sam
$ tree
.
├── README.md
├── __init__.py
├── events
│   └── event.json
├── hello_world
│   ├── __init__.py
│   ├── app.py
│   └── requirements.txt
├── template.yaml
└── tests
    ├── __init__.py
    └── unit
        ├── __init__.py
        └── test_handler.py

The file named template.yaml in this directory is important.
In AWS SAM, you define AWS SAM resources by creating a configuration file called template.yaml.

The content of template.yaml looks like this:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  hello-sam

  Sample SAM Template for hello-sam

# More info about Globals: https://github.com/awslabs/serverless-application-model/blob/master/docs/globals.rst
Globals:
  Function:
    Timeout: 3

Resources:
  HelloWorldFunction:
    Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
    Properties:
      CodeUri: hello_world/
      Handler: app.lambda_handler
      Runtime: python3.7
      Events:
        HelloWorld:
          Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api
          Properties:
            Path: /hello
            Method: get

Outputs:
  # ServerlessRestApi is an implicit API created out of Events key under Serverless::Function
  # Find out more about other implicit resources you can reference within SAM
  # https://github.com/awslabs/serverless-application-model/blob/master/docs/internals/generated_resources.rst#api
  HelloWorldApi:
    Description: "API Gateway endpoint URL for Prod stage for Hello World function"
    Value: !Sub "https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/hello/"
  HelloWorldFunction:
    Description: "Hello World Lambda Function ARN"
    Value: !GetAtt HelloWorldFunction.Arn
  HelloWorldFunctionIamRole:
    Description: "Implicit IAM Role created for Hello World function"
    Value: !GetAtt HelloWorldFunctionRole.Arn

AWS SAM works by running CloudFormation internally to create AWS resources based on the configuration file you wrote as code.
In the Resources section of template.yaml, we define the Lambda and API Gateway we are building this time.

CircleCI Preparation

Integration with GitHub Repository

Since CircleCI can run automatic deployments when code is pushed to a GitHub repository, we need to link the corresponding repository with CircleCI in advance.
(If you are signed into CircleCI with your GitHub account, they should already be linked as shown below.)

Additionally, the information for the GitHub repository you created beforehand should be linked.
This time, I created a repository named "circleci-sam-orb", so I select [Set Up Project].

While you're at it, execute [Start Building] so you don't forget.

Now, a build will automatically trigger whenever code is pushed to the main branch of the repository.

Setting Environment Variables

When deploying to AWS from CircleCI, you will use the AWS access key and secret key you prepared earlier, but this information must never be leaked.
In such cases, CircleCI allows you to register those values as environment variables, which is convenient because you can use them whenever needed.

You can set environment variables for each Project on CircleCI, but I'll try using the Contexts feature, which allows you to set environment variables across multiple Projects.
(Reference: Using Contexts)

On the CircleCI console, select [Organization Settings] → [Contexts] → [Create Context].
Here, you create a set of environment variables, and you'll be able to call those environment variables using the Context Name.
This is recommended because it makes it easier to partially reuse configuration files.

I'll set the Context Name to "aws-credentials-context" to match the official Orbs documentation.
Once entered, select [Create Context].

Once created, select [Add Environment Variable] within aws-credentials-context and set the following three environment variables:

• AWS_ACCESS_KEY_ID: The value you noted down
• AWS_SECRET_ACCESS_KEY: The value you noted down
• AWS_DEFAULT_REGION: The region you want to use (e.g., "ap-northeast-1" for Tokyo)

When finished, the screen should look like this.

Creating S3 Buckets

Create the S3 buckets that will be used for uploading related files during automated testing and deployment by CircleCI.
This time, I created the following two buckets, but please replace them with the bucket names you created in your AWS account as appropriate.

• For testing: circleci-sam-orb-test
• For deployment: circleci-sam-orb-prod

Creating the CircleCI Configuration File

In CircleCI, CI/CD operations are based on a configuration file named .circleci/config.yml created in the root directory of the project.
Therefore, please create a directory named .circleci inside the hello-sam directory, and then create an empty file named config.yml within it.

Now, referring to the build_test_deploy example from the Orbs official documentation, we will write the contents of config.yml as follows:

hello-sam/.circleci/config.yml

version: '2.1'
orbs:
  sam: circleci/aws-sam-serverless@3.0
jobs:
  test_my_api:
    docker:
      - image: 'cimg/node:lts'
    steps:
      - run: echo "Run your API tests here"
workflows:
  test_and_deploy:
    jobs:
      - sam/deploy:
          context: aws-credentials-context
          name: deploy-staging
          s3-bucket: circleci-sam-orb-test
          stack-name: staging-stack
          template: ./template.yaml
      - test_my_api:
          requires:
            - deploy-staging
      - sam/deploy:
          context: aws-credentials-context
          name: deploy-production
          s3-bucket: circleci-sam-orb-prod
          requires:
            - test_my_api
          stack-name: production-stack
          template: ./template.yaml

The key points are specifying the aws-credentials-context we created earlier as the Context, and specifying the S3 buckets created for the test and deployment phases respectively as the s3-bucket.

Once you've done this, the preparation is complete.

Executing Automated Deployment

Push all code to the GitHub repository

By default, CircleCI detects updates to the code in the main branch and triggers CI/CD.
Specify the branch as main and push the code along with .circleci/config.yml to the repository you prepared.

$ git add .
$ git commit -m "first commit"
$ git branch -M main
$ git push -u origin main

Once successfully pushed, it will handle everything up to automated deployment automatically.

Check on the CircleCI Dashboard

You can check the job status from the [Dashboard] on the CircleCI console.

As time passes, the job status will change, and if all jobs are completed without issues, they will be displayed as "Success" with green checkmarks.

Call the Deployed API

Now let's actually call the deployed API.
We'll access the URL using the curl command.

On CircleCI, select [deploy-production] from the previous deployment results.

A list of each step will be displayed, so select and expand [Deploy SAM application].

From here, it's the domain of AWS SAM rather than CircleCI, but the same results as when running the "sam deploy" command are displayed here (in fact, that's how it works internally).
The API Gateway endpoint URL should be displayed in the HelloWorldAPI field within Outputs, so make a note of it.

Note that the item displayed as "************" contains the region name where the API Gateway was deployed, but since it is a value set as an environment variable in CircleCI, it is automatically masked.
In this case, we specified "ap-northeast-1", so let's supplement it ourselves.

Let's access this URL using the curl command.

$ curl https://uatlffehm2.execute-api.ap-northeast-1.amazonaws.com/Prod/hello/
{"message": "hello world"}

I confirmed that the message was returned in JSON format as specified in the AWS SAM HelloWorld template.
The deployment is successfully complete! Great job.

Conclusion

I realized that using CircleCI Orbs allows for a smaller amount of configuration in config.yml, making the overall deployment much simpler.
In fact, looking at the source of the Orb I used this time, the config.yml is over 500 lines long, so the fact that it can be handled in about 30 lines is very powerful.

To be honest, I was a bit worried that the internal workings might be a black box, making it difficult to respond to errors, but since you can check the contents within each Orb's documentation, troubleshooting while reading the execution results was easy.
(In fact, while I was trying it out, an error occurred related to S3 bucket settings, but I was able to troubleshoot it without any problems.)

In my personal opinion, people using CircleCI for the first time might find it beneficial to start with Orbs where the amount of configuration is small, and gradually customize things as they want to do more.
Since it's a convenient set provided officially, let's take advantage of it.