Using AWS SDK for JavaScript v3 with Google Cloud Storage

Introduction

Good morning, I'm Kato. Recently, there's been a lot of buzz around object storage services like Wasabi and Cloudflare R2. These services, along with Google Cloud's Cloud Storage (the main focus here), are said to be compatible with Amazon S3.

While I've known about this for a while, I was curious if I could actually leverage this compatibility. Specifically, I wanted to see how many changes would be needed to use Cloud Storage with code written using the AWS SDK, so I decided to give it a try.

Researching the Documentation

First, I searched for information online and found the relevant official documentation: Migrating from Amazon S3 to Cloud Storage | Google Cloud
Unfortunately, there was no sample code for JavaScript, but I gathered the following information from this page:

• There are two approaches to using Cloud Storage while continuing to use the AWS SDK: Simple Migration and Full Migration.
• Simple migration has limitations on available features (OAuth 2.0, access control).
• Simple migration uses HMAC keys for authentication.

In this article, I will try out the simple migration approach.

What is an HMAC Key?

HMAC keys | Cloud Storage | Google Cloud

An HMAC key is a credential consisting of an access ID and a secret. It seems to be used in limited cases, such as when you want to reuse existing code to access Cloud Storage, as in this instance.
You can create a service account with permissions for Cloud Storage operations and issue an HMAC key for it. In other words, it is the equivalent of an Access Key and Secret Access Key in AWS. As mentioned earlier, please note that this is intended for limited use cases.

Operating Cloud Storage with AWS SDK for JavaScript v3

I actually tried writing the code. First, follow the steps in the documentation to create a service account and an HMAC key.
Create an HMAC key

For this exercise, it is sufficient to grant permissions to list and get buckets.

Here is the excerpted code:

async function listBuckets(client: S3): Promise<string[]> {
  const res = await client.send(new ListBucketsCommand({}));

  if (res.Buckets === undefined) {
    return [];
  }

  return res.Buckets.flatMap((bucket) =>
    bucket.Name === undefined ? [] : [bucket.Name]
  );
}

async function getObject(
  client: S3,
  bucket: string,
  key: string
): Promise<string> {
  const res = await client.send(
    new GetObjectCommand({ Bucket: bucket, Key: key })
  );

  return streamToString(res.Body);
}

function cloudStorage(): S3 {
  return new S3({
    credentials: {
      accessKeyId: process.env.GOOGLE_ACCESS_KEY_ID!,
      secretAccessKey: process.env.GOOGLE_ACCESS_KEY_SECRET!,
    },
    endpoint: "https://storage.googleapis.com",
  });
}

async function main(): Promise<void> {
  const gcs = cloudStorage();

  console.log(await listBuckets(gcs));
  console.log(await getObject(gcs, process.env.BUCKET!, process.env.KEY!));
}

By setting the HMAC key in the credentials and specifying the endpoint during client instantiation, you can use it as a Cloud Storage client. While I only tested listing and getting objects this time, according to the documentation, it seems that full CRUD operations can be performed without restrictions.

Afterword

Actually, in a recent project, I performed a task that was the opposite of this: replacing code that originally used Cloud Storage with S3. Since the project's codebase was not large, I swapped the SDK and rewrote the code, but I was curious about how it would have turned out if I had utilized S3 compatibility instead, which is why I looked into it.

Given that proper documentation is available, it seems to be an intended use case and should be usable at a production level without any issues. In most cases, I think people would switch to the Google Cloud SDK. However, I hope the method introduced here will be helpful in situations where you want to migrate gradually or based on other specific needs.