Various Ways to Deploy NestJS to Cloud Run and Setting Up Continuous Deployment

Introduction

NestJS is a server-side TypeScript framework that I've been keeping an eye on. While tinkering with it locally, I felt like I wanted to deploy it. There are various deployment options, but since I've been working with Google Cloud lately, I decided to try deploying it to Cloud Run. I'll record the deployment process in this article.

Conclusive Opinion

To deploy NestJS (or rather, a Node.js app) to Cloud Run, you have the following options:

1. Prepare only the source code and leave everything to gcloud run deploy.
2. Prepare the source code and a Dockerfile, then leave the rest to gcloud run deploy.
3. Define the source code, Dockerfile, and Cloud Build triggers to create revisions from Artifact Registry at any timing.

For personal development, option 1 or 2 is sufficient. While option 1 handles everything like magic, the contents of the image are such a black box that it was difficult to reproduce errors when they occurred. Therefore, I recommend the approach of preparing a Dockerfile as in option 2. Option 3 is useful when you want fine-grained control over the timing of container creation, Cloud Run revision creation, and switching traffic to the new revision for applications running as a service.

What exactly is happening during a Cloud Run deployment?

Looking at the Cloud Run deployment page, the categories are:

• Deploying container images
• Continuous deployment from git
• Deploying from source code

However, I feel these categories are a bit simplified. If written without omissions, no matter the method, it follows a flow like this:

If you want to know more about what happens during deployment, this blog post is a great resource.

https://qiita.com/sakas1231/items/423cd7efcce77f5e426a

In other words, from the perspective of Cloud Run, it means as long as you provide a container image, you can deploy. However, for users who might feel that preparing a container image is a bit of a challenge, the gcloud run deploy command offers various deployment methods.

Re-classifying the deployment methods

Based on what we've discussed so far, let's organize the deployment methods once more.

1. Prepare only the source code and leave everything to gcloud run deploy.
2. Prepare the source code and a Dockerfile, then leave the rest to gcloud run deploy.
3. Define the source code, Dockerfile, and Cloud Build triggers to create revisions from Artifact Registry at any timing.

To reiterate, the deployment process itself—Create container image => Create Cloud Run revision using it => Direct traffic to the new revision—remains the same. The only difference is who performs these steps and when.

Common preparation for deployment

First, let's adjust the NestJS application so that it can be deployed. This step is common for deploying to Cloud Run, regardless of the method you choose. After completing the initial setup using something like yarn nest new, modify main.ts as follows:

main.ts

import { NestFactory } from '@nestjs/core';
import { AppModule } from './app.module';
import {
  FastifyAdapter,
  NestFastifyApplication,
} from '@nestjs/platform-fastify';

async function bootstrap() {
  const app = await NestFactory.create<NestFastifyApplication>(
    AppModule,
    new FastifyAdapter({ logger: true }),
  );
+  const port = Number(process.env.PORT) || 3000; // Cloud Run requirement. Ensure it starts using the PORT environment variable.
+  await app.listen(port, '0.0.0.0'); // Add '0.0.0.0' to accept external access.
}

bootstrap();

Next, prepare an endpoint to verify connectivity after deployment. It doesn't necessarily have to be the root.

yarn start
curl "http://localhost:3000/present"
[{"name":"happy_birthday_pack","prize":1},{"name":"good","prize":2}]

That's it for the application modifications.

From here on, we'll be using gcloud commands. This assumes that you've already authenticated with the Google Cloud project where you'll be deploying. You can perform the authentication as follows:

gcloud config configurations activate my-project
gcloud auth login

Leaving everything to the command with only source code (Failed to start)

I haven't been able to resolve the "Container terminated due to sandbox error" error that occurs during revision creation, so it's not starting. If anyone knows anything about this, please let me know. For now, I've given up on the "leave-everything-to-the-command" course for NestJS. 🙇‍♂️

To deploy from source code, you need to execute the build beforehand and create the dist directory. Looking into it, the breakdown of responsibilities seems to be as follows:

1. Build the source code (Developer)
2. Prepare .gcloudignore (Developer)
3. Execute gcloud run deploy (Developer)
4. Upload source code to GCS (gcloud)
5. Cloud Build trigger starts
   1. Runtime detection
   2. yarn install --production
   3. Image build
   4. Push the built image to Artifact Registry
6. Create Cloud Run revision using the image from Artifact Registry (gcloud)
7. Direct 100% of traffic to the new revision (gcloud)

Steps up to 3 are performed by us. You might wonder, "How is the image being built without a Dockerfile...?" Apparently, it uses a Google tool called Buildpacks to automatically detect the runtime and build the image. That's quite an impressive technology.

1. Build the source code

Probably the part where you collect the materials needed for the build should be done in the developer's environment beforehand. So, let's run it locally.

yarn build

If the dist directory is created, it's okay.

2. Prepare .gcloudignore

I initially struggled because I didn't do this... If .gcloudignore doesn't exist, the gcloud run deploy command will refer to .gitignore to try and keep things slim. .gitignore usually excludes dist, meaning the build artifacts won't be uploaded to the Cloud Storage bucket. Since we want to include dist as an upload target this time, we'll prepare a .gcloudignore file.

.gcloudignore

# compiled output
/node_modules

# Logs
logs
*.log

# OS
.DS_Store
...

It's fine as long as dist isn't included here. Conversely, make sure to exclude node_modules. It takes time to upload to GCS, and since yarn install is performed during the image build process, uploading them is unnecessary.

3. Execute gcloud run deploy

gcloud run deploy \ 
  backend-user-from-source \ # 1
  --region asia-northeast2 \
  --command="yarn start:prod" \ # 2
  --source . # 3

1. This is the Cloud Run service name. Its meaning is almost the same as an App Engine service. At the risk of sounding too simple, you could call it the microservice name.
2. Since docker build is executed in the background, you specify the CMD for it. If nothing is specified, it defaults to yarn start, but since NestJS provides yarn start:prod for production use, we specify that here.
3. Executes using the current directory as the upload target.

Now we just wait!

Building using Buildpacks and deploying container to Cloud Run service [backend-user] in project [my-project] region [asia-northeast2]
X Building and deploying... Cloud Run error: Internal error occurred while performing container health check.
  ✓ Uploading sources...
  ✓ Building Container... Logs are available at [https://console.cloud.google.com/cloud-build/builds/xxx].
  X Creating Revision... Cloud Run error: Internal error occurred while performing container health check.
  . Routing traffic...
Deployment failed
ERROR: (gcloud.run.deploy) Cloud Run error: Internal error occurred while performing container health check.

Failure. It seems like creating the revision failed... Let's look at the logs...

Container terminated due to sandbox error. appeared, and I couldn't trace it any further. Note that when I pulled the image uploaded to Artifact Registry and ran yarn start:prod locally, it started normally. There was nothing obviously wrong with the container image, so there's not much I can do on my end. Does anyone know why this is? 🤔

Prepare a Dockerfile and leave the rest to the command

For personal development, this method seemed to offer the best value for the effort. The breakdown of responsibilities is as follows:

1. Prepare .gcloudignore (Developer)
2. Prepare a Dockerfile (Developer)
3. Execute gcloud run deploy (Developer)
4. Upload source code to GCS (gcloud)
5. Build the image using the Dockerfile (Cloud Build)
6. Push the built image to Artifact Registry (Cloud Build)
7. Create a Cloud Run revision using the image from Artifact Registry (gcloud)
8. Direct 100% of traffic to the new revision (gcloud)

In this method, since a Dockerfile is provided, the automatically triggered Cloud Build will build the image using that Dockerfile instead of using Buildpacks.

1. Prepare .gcloudignore

The same file as before can be used. I'll omit the details here.

2. Prepare a Dockerfile

I prepared the following file. It uses a multi-stage build, separating the stages for building and running.

# syntax=docker/dockerfile:1
FROM node:16 AS builder
# Because devDependencies also need to be installed for the build
ENV NODE_ENV=development
WORKDIR /app
COPY package.json ./
COPY yarn.lock ./
RUN yarn install
COPY . .
RUN yarn build


FROM node:16-stretch-slim AS runner
ARG NODE_ENV=production
ENV NODE_ENV=${NODE_ENV}
WORKDIR /app
COPY package.json ./
COPY yarn.lock ./
# When running yarn install (npm install) with NODE_ENV=production, devDependencies are not installed
RUN yarn install
COPY --from=builder /app/dist ./dist
CMD ["yarn", "start:prod"]

References:
https://matsuand.github.io/docs.docker.jp.onthefly/develop/develop-images/multistage-build/
https://times.hrbrain.co.jp/entry/2020/12/17/nestjs-docker-optimize
https://dev.to/erezhod/setting-up-a-nestjs-project-with-docker-for-back-end-development-30lg

3. Execute gcloud run deploy

Since the startup command is specified in the CMD, there is no longer a need to pass it via gcloud options.

gcloud run deploy \ 
  backend-user-from-source \
  --region asia-northeast2 \
  --source .

✓ Building and deploying... Done.
  ✓ Uploading sources...
  ✓ Building Container...
  ✓ Creating Revision...
  ✓ Routing traffic...

Successfully deployed. The wait time was about 5 minutes. Let's also verify connectivity.

yarn start

curl "https://backend-user-xxxxxxx-dt.a.run.app/present"
[{"name":"happy_birthday_pack","prize":1},{"name":"good","prize":2}]

It seems to be working fine. While creating a Dockerfile takes a bit of effort, I feel this method offers the best balance between effort and convenience.

Continuous Deployment using Cloud Build

Let's consider the application deployment cycle. If it's enough just to always have the latest code deployed, the methods introduced earlier would be fine. However, when updating a running application, for example, the shorter the release time, the better. In other words, suppose we want to fulfill a requirement where we create the Cloud Run revision before the actual release and then only switch the traffic at the time of release.

0. Prepare a Dockerfile (Developer)
1. Create an Artifact Registry repository in advance (Developer)
2. Prepare cloudbuild.yaml (Developer)
3. Create a Cloud Build GitHub trigger that starts on a push to the main branch (Developer)
4. Cloud Build starts (Automatic)
   1. Build the image using the Dockerfile
   2. Push the built image to Artifact Registry
   3. Create a Cloud Run revision using the Artifact Registry image
5. Direct 100% of traffic to the new revision (Developer)

After creating the trigger once, you will be able to run continuous deployment through steps 4 and 5.

Let's look at this in detail.

0. Prepare a Dockerfile

The same one as before can be used. I'll include it again.

# syntax=docker/dockerfile:1
FROM node:16 AS builder
# Because devDependencies also need to be installed for the build
ENV NODE_ENV=development
WORKDIR /app
COPY package.json ./
COPY yarn.lock ./
RUN yarn install
COPY . .
RUN yarn build


FROM node:16-stretch-slim AS runner
ARG NODE_ENV=production
ENV NODE_ENV=${NODE_ENV}
WORKDIR /app
COPY package.json ./
COPY yarn.lock ./
# When running yarn install (npm install) with NODE_ENV=production, devDependencies are not installed
RUN yarn install
COPY --from=builder /app/dist ./dist
CMD ["yarn", "start:prod"]

1. Create an Artifact Registry repository in advance

While the "leave-it-all-to-the-command" course automatically creates the Artifact Registry repository, here the developer must create it. The creation itself can be done via gcloud or the console, but I recommend applying the infrastructure management method defined by your organization or team. This time, I created it using Terraform.

https://cloud.google.com/artifact-registry/docs/integrate-terraform?hl=ja

You can create an Artifact Registry repository with content almost exactly as described there. I created a module like the following.

modules/artifact-registry/main.ts

variable "artifact_registry_location" {
  type = string
  # https://cloud.google.com/storage/docs/locations
  description = "Location for the Artifact Registry"
}


resource "google_artifact_registry_repository" "gql-nest-prisma-training" {
  provider = google-beta

  location      = var.artifact_registry_location
  repository_id = "gql-nest-prisma-training"
  description = "Backend API for users"
  format = "DOCKER"
}

Prepare this file and run:

terraform apply

This will create the repository.

Regarding repository_id = "gql-nest-prisma-training": this becomes the repository name. You can choose any name for the Artifact Registry repository, but it seems safest to match it with your GitHub repository name.

2. Prepare cloudbuild.yaml

https://cloud.google.com/build/docs/deploying-builds/deploy-cloud-run?hl=ja#required_iam_permissions

I used this as a reference. First, grant permissions to the Cloud Build service account. Specifically, the Cloud Run Admin role. I did this through the console.

Next, I created the cloudbuild.yaml file.

It's based on this, but with the difference that it pushes to Artifact Registry (the sample uses Container Registry).

deploy/cloudbuild.yaml

steps:
  # 4-1. Build the container image
  - name: 'gcr.io/cloud-builders/docker'
    args: ['build', '-t', '${_REGION}-docker.pkg.dev/$PROJECT_ID/${_REPOSITORY}/${_SERVICE_NAME}:$SHORT_SHA', '.' ]
  # 4-2. Push the container image to Container Registry
  - name: 'gcr.io/cloud-builders/docker'
    args: ['push', '${_REGION}-docker.pkg.dev/$PROJECT_ID/${_REPOSITORY}/${_SERVICE_NAME}:$SHORT_SHA']
  # 4-3. Create Cloud Run Revision using image
  - name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
    entrypoint: gcloud
    args:
      - 'run'
      - 'deploy'
      - ${_SERVICE_NAME}
      - '--no-traffic'
      - '--image'
      - '${_REGION}-docker.pkg.dev/$PROJECT_ID/${_REPOSITORY}/${_SERVICE_NAME}:$SHORT_SHA'
      - '--region'
      - '$_REGION'
      - '--revision-suffix'
      - $SHORT_SHA
  # 4-4. Display traffic update command
  - name: 'bash'
    entrypoint: 'echo'
    args:
      - 'gcloud run services update-traffic ${_SERVICE_NAME} --region=${_REGION} --to-revisions=${_SERVICE_NAME}-$SHORT_SHA=100'

substitutions:
  _REGION: asia-northeast2
  _REPOSITORY: $REPO_NAME
  _SERVICE_NAME: backend-user

• 4-1: I'm building the image with a tag because it will be pushed later.
• 4-2: Pushing to Artifact Registry.
• 4-3: Creating the Cloud Run revision with 0% traffic.
• 4-4: Since this trigger has context such as the short SHA, I display the traffic migration command for convenience (for copy-pasting).

Everything up to this point can be executed just before release. Now, let's actually run the trigger.

When the Cloud Build trigger is activated, cloudbuild.yaml is located from the source code, and the build proceeds.

You can see that the image is being pushed to Artifact Registry 👇

Using this image, a new revision has been created in Cloud Run 👇

The job of the Cloud Build trigger ends here. Finally, at the time of release, developers can copy and paste the migration command in Cloud Shell or a similar environment to execute it.

Traffic has been successfully migrated to the new revision. The traffic switch completes in about 5 seconds. If there are any issues, you can just revert the traffic to the previous revision.

Conclusion

I've explored various patterns for deploying NestJS to Cloud Run. Setting up a continuous deployment mechanism from the start makes it easier to gradually improve the deployment flow later on. As mentioned at the beginning:

• Personal development level: Create only the Dockerfile and leave the rest to the command
• Production products: Create Cloud Build triggers and limit release tasks to traffic migration only

I found this to be a cost-effective approach.

Since I plan to use NestJS in conjunction with Prisma, I'll try deploying a combination of Cloud Run and Cloud SQL next.