Open5ヶ月前にコメント追加2

Cloud Functions 2nd gen + Node で Cloud Storage 間のファイル転送

Node.js

Cloud Functions

Cloud Storage

tokadev

オブジェクト作成をトリガーに bucket 間でオブジェクトをコピーするために、 Cloud Functions (2nd gen) + Node を使ってみたのでその備忘録。
（といってもほぼドキュメントのサンプルコードまま

ディレクトリ構成

objectCopy/
├ index.js
└ package.json

index.js

/**
 * * Copies the file to the other bucket, when the object created.
 */
const functions = require('@google-cloud/functions-framework');
const { Storage } = require('@google-cloud/storage');

// コピー先の CloudStorage(Bucket) 名
const bucketNameTo = process.env.BUCKET_NAME_TO;

functions.cloudEvent('finalized', cloudEvent => {
  const object = cloudEvent.data;
  objectCopy(object.bucket, object.name, bucketNameTo, object.name, 0);
});

function objectCopy(
  srcBucketName,  // コピー元の bucket
  srcFilename,    // コピー元の ファイル名
  destBucketName, // コピー先の bucket
  destFileName,   // コピー先の ファイル名
  destinationGenerationMatchPrecondition = 0
) {
  const storage = new Storage();

  async function objectCopyToOtherBucket() {
    const copyDestination = storage.bucket(destBucketName).file(destFileName);

    const copyOptions = {
      preconditionOpts: {
        ifGenerationMatch: destinationGenerationMatchPrecondition,
      },
    };

    try {
      await storage
        .bucket(srcBucketName)
        .file(srcFilename)
        .copy(copyDestination, copyOptions);
      console.log(`${srcBucketName}/${srcFilename} to ${destBucketName}/${destFileName} success.`);

    } catch (e) {
      // severity を error に設定して Logging
      console.error(new Error(`${srcBucketName}/${srcFilename} to ${destBucketName}/${destFileName} failed.`));
    }
  }

  objectCopyToOtherBucket().catch(console.error);
}

オブジェクト作成の検知には CloudEvent 関数を使用

process.env.VAR_NAME の VAR_NAME の部分は Cloud Functions のランタイム環境変数

デプロイに使う package.json

package.json

{
  "name": "objectcopy",
  "version": "1.0.0",
  "main": "index.js",
  "license": "Apache-2.0",
  "engines": {
    "node": ">=20.0.0"
  },
  "dependencies": {
    "@google-cloud/functions-framework": "^3.4.0",
    "@google-cloud/storage": "^7.11.2",
    "typescript": "^5.4.5"
  }
}

tokadev

デプロイ手順

デプロイ前準備

cd objectCopy;
yarn install;

gcloud auth login;

環境変数設定

$FUNCTION_NAME はお好みで。

PROJECT_ID=$(gcloud config get-value project);
PROJECT_NUMBER=$(gcloud projects list --filter="project_id:$PROJECT_ID" --format='value(project_number)');
SERVICE_ACCOUNT=$(gsutil kms serviceaccount -p $PROJECT_NUMBER);
BUCKET_NAME_FROM=gs://bucket-from;
BUCKET_NAME_TO=gs://bucket-to;
FUNCTION_NAME=deploy-cloud-functions-name;

gcloud config set project $PROJECT_ID;
gcloud components update;

権限付与

CloudStorage の finalize を CloudFunctions トリガーに設定する場合、Pub/Sub と EventArc の権限も必要になるため付与しておく。
また、コピー先のプロジェクトの CloudStorage への権限も別途必要になるので忘れずに。

## Pub/Sub パブリッシャー
gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member serviceAccount:$SERVICE_ACCOUNT \
  --role roles/pubsub.publisher;
## EventArc イベント受信者
gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member serviceAccount:$SERVICE_ACCOUNT \
  --role roles/eventarc.eventReceiver;
## コピー先の bucket に対して Storage オブジェクト作成者
gcloud storage buckets add-iam-policy-binding $BUCKET_NAME_TO \
  --member serviceAccount:$SERVICE_ACCOUNT \
  --role roles/storage.objectCreator;

権限付与しないでデプロイした場合のエラー

# Pub/Sub
ERROR: (gcloud.functions.deploy) OperationError: code=7, message=Creating trigger failed for projects/{project_id}/locations/asia-northeast1/triggers/{trigger_id}: The Cloud Storage service account for your bucket is unable to publish to Cloud Pub/Sub topics in the specified project.
To use GCS CloudEvent triggers, the GCS service account requires the Pub/Sub Publisher (roles/pubsub.publisher) IAM role in the specified project. (See https://cloud.google.com/eventarc/docs/run/quickstart-storage#before-you-begin): permission denied

# EventArc
ERROR: (gcloud.functions.deploy) ResponseError: status=[400], code=[Ok], message=[Validation failed for trigger projects/{project_id}/locations/asia-northeast1/triggers/{trigger_id}: Invalid resource state for "": Permission denied while using the Eventarc Service Agent. If you recently started to use Eventarc, it may take a few minutes before all necessary permissions are propagated to the Service Agent. Otherwise, verify that it has Eventarc Service Agent role.]

CloudFunctions デプロイ

今回は node でデプロイ、 asia-northeast1 は東京リージョン。
CloudFunctions のランタイム変数を使う場合は --set-env-vars オプションで設定する。

gcloud functions deploy $FUNCTION_NAME \
  --gen2 \
  --runtime=nodejs20 \
  --region=asia-northeast1 \
  --source=. \
  --entry-point=finalized \
  --trigger-bucket=$BUCKET_NAME_FROM \
  --trigger-service-account=$SERVICE_ACCOUNT \
  --set-env-vars BUCKET_NAME_TO=$BUCKET_NAME_TO \
  --allow-unauthenticated;