iTranslated by AI

The content below is an AI-generated translation. This is an experimental feature, and may contain errors. View original article
😫

How to Resolve CannotPullContainerError When Deploying to Fargate from Apple Silicon Macs

に公開

Introduction

When you build a Docker image on an Apple Silicon Mac (M1/M2/M3/M4), push it to Amazon ECR, and attempt to run it on ECS Fargate, the following error may occur:

CannotPullContainerError: pull image manifest has been retried 7 time(s):
image Manifest does not contain descriptor matching platform 'linux/amd64'

This error is caused by a mismatch between the CPU architecture of the container image and the ECS Fargate execution environment. However, there are cases where simply adding --platform linux/amd64 to the build command does not solve the problem.

In this article, I will explain why this issue occurs, including the internal mechanism of Docker Desktop, and introduce a reliable solution.

Target Audience

  • Those who want to deploy Amd64 images to ECS Fargate from an Apple Silicon environment.
    • If you want to run Arm on ECS in the first place, you can change the runtime_platform to Arm in the ECS Fargate task definition.
    • I chose amd to match the existing amd environment, but if there are no specific issues, switching to arm is also a good option.

Prerequisites

Item Value
Local Machine Apple Silicon Mac (M1/M2/M3/M4). The author uses an M4 Max.
Docker Docker Desktop for Mac (BuildKit / buildx enabled by default)
Container Registry Amazon ECR
Container Runtime Environment Amazon ECS Fargate (Default: X86_64)

Background Knowledge

Differences in CPU Architectures

Container images are built for specific CPU architectures. It is important to understand these two main architectures:

Architecture Alias Main Usage Environments
x86_64 amd64 Intel/AMD CPUs, AWS Fargate (default), most cloud infrastructures
ARM64 aarch64 Apple Silicon Mac, AWS Graviton, Raspberry Pi

Differences between Apple Silicon Mac and Intel Mac

  • Intel Mac: x86_64 (amd64) architecture. Built images work on ECS Fargate as they are.
  • Apple Silicon Mac: ARM64 architecture. Building by default generates an ARM64 image, which will not run on ECS Fargate (x86_64).

Default Architecture of ECS Fargate

If runtime_platform is not specified in the ECS Fargate task definition, X86_64 (amd64) is used by default.

# If runtime_platform is not specified in the task definition, X86_64 is used by default.
resource "aws_ecs_task_definition" "webapp" {
  family                   = "webapp-task"
  requires_compatibilities = ["FARGATE"]
  network_mode             = "awsvpc"
  cpu                      = "512"
  memory                   = "1024"
  # runtime_platform is omitted -> Defaults to X86_64
  # ...
}

Why This Problem Occurs

Internal Behavior of Docker Desktop on Apple Silicon

In Docker Desktop on Apple Silicon Mac, the docker build command uses buildx (BuildKit) internally. This results in the image being generated in the OCI Image Index format instead of the traditional Docker V2 manifest.

This is the root cause of the problem.

What is OCI Image Index?

OCI Image Index is a manifest format for managing images for multiple architectures under a single tag.

When you perform docker build on an Apple Silicon Mac, an OCI Image Index is generated, which contains the ARM64 image and an attestation manifest. The amd64 image is not included.

Comparison of Failure and Success Patterns

The Essence of the Problem

When you push using the two-step process of docker build + docker push, it follows this path:

  1. docker build → BuildKit generates an OCI Image Index (arm64 + attestation)
  2. Stores the image in the local Docker daemon
  3. docker push → Sends the OCI Image Index to ECR as is

Even if you write FROM --platform=linux/amd64 in your Dockerfile, what docker push sends is the OCI Image Index stored in the local Docker daemon, and its platform description remains arm64.


What actually happened

Appears as amd64 locally

Checking the local image with docker inspect shows amd64.

$ docker inspect <image>:latest | grep Architecture
        "Architecture": "amd64",

However, checking the manifest on ECR reveals that it is registered as arm64.

How to check the manifest on ECR

aws ecr batch-get-image \
  --repository-name <repository-name> \
  --image-ids imageTag=latest \
  --query 'images[0].imageManifest' \
  --output text | python3 -m json.tool

Manifest when registered as arm64

As shown below, only arm64 was included in the OCI Image Index format.

{
    "schemaVersion": 2,
    "mediaType": "application/vnd.oci.image.index.v1+json",
    "manifests": [
        {
            "mediaType": "application/vnd.oci.image.manifest.v1+json",
            "digest": "sha256:b298d25f...",
            "size": 1624,
            "platform": {
                "architecture": "arm64",
                "os": "linux"
            }
        },
        {
            "mediaType": "application/vnd.oci.image.manifest.v1+json",
            "digest": "sha256:0ab29bc6...",
            "size": 566,
            "annotations": {
                "vnd.docker.reference.type": "attestation-manifest"
            },
            "platform": {
                "architecture": "unknown",
                "os": "unknown"
            }
        }
    ]
}

Key points:

  • "architecture": "arm64" → Does not work on ECS Fargate (amd64)
  • "architecture": "unknown" → Attestation manifest (provenance information, not the image itself)
  • No amd64 entry exists

Healthy manifest (amd64)

After applying the fix, it became a Docker V2 Manifest (single architecture) format.

{
    "schemaVersion": 2,
    "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
    "config": {
        "mediaType": "application/vnd.docker.container.image.v1+json",
        "digest": "sha256:5c7b82d8...",
        "size": 7428
    },
    "layers": [
        { "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", "digest": "sha256:...", "size": 3861821 },
        { "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", "digest": "sha256:...", "size": 51601924 }
    ]
}

Key points:

  • "mediaType": "application/vnd.docker.distribution.manifest.v2+json" → Docker V2 format (not OCI Index)
  • layers array instead of manifests array → Single architecture image

Solution

Build & Push Command

docker buildx build \
  --platform linux/amd64 \
  --provenance=false \
  -t <ECR Repository URL>:latest \
  --push \
  ./app

Role of each option:

Option Role
--platform linux/amd64 Specifies the build target as amd64. Generates an amd64 image through cross-compilation even on Apple Silicon Macs.
--provenance=false Disables the generation of the attestation (provenance information) manifest. This causes the image to be in Docker V2 Manifest format instead of OCI Image Index format.
--push Pushes directly to the registry from buildx. Since it bypasses the local Docker daemon, manifest inconsistency does not occur.

If You Have Already Pushed to ECR

If an old arm64 image remains in ECR, the manifest might not be updated due to layer caching, resulting in Layer already exists. Please delete it once and then re-push.

# 1. Delete the existing image in ECR
aws ecr batch-delete-image \
  --repository-name <repository-name> \
  --image-ids imageTag=latest

# 2. Also delete remaining images without tags
aws ecr list-images \
  --repository-name <repository-name> \
  --query 'imageIds[*]' \
  --output json | xargs -I{} \
  aws ecr batch-delete-image \
  --repository-name <repository-name> \
  --image-ids '{}'

# 3. Rebuild & Push
docker buildx build \
  --platform linux/amd64 \
  --provenance=false \
  -t <ECR Repository URL>:latest \
  --push \
  ./app

# 4. Confirm the manifest
aws ecr batch-get-image \
  --repository-name <repository-name> \
  --image-ids imageTag=latest \
  --query 'images[0].imageManifest' \
  --output text | python3 -m json.tool

Forcing ECS Redeployment

Simply updating the image in ECR may not be enough as ECS might have the old image cached. Use forced redeployment to pull the new image.

aws ecs update-service \
  --cluster <cluster-name> \
  --service <service-name> \
  --force-new-deployment

Checking deployment status:

aws ecs describe-services \
  --cluster <cluster-name> \
  --services <service-name> \
  --query 'services[0].{runningCount:runningCount,deployments:deployments[*].{status:status,runningCount:runningCount,rolloutState:rolloutState}}'
  • runningCount: 1 + rolloutState: "COMPLETED" → Deployment successful
  • rolloutState: "IN_PROGRESS" → Still in progress
  • rolloutState: "FAILED" → Failed (Check CloudWatch Logs)

Common Ineffective Fixes

NG 1: Addressing with just FROM --platform=linux/amd64

# This is not enough
FROM --platform=linux/amd64 node:22-alpine

Specifying --platform in the Dockerfile pulls the amd64 variant of the base image. However, the platform description in the OCI Image Index generated by Docker Desktop does not change. The manifest sent to ECR via docker push still references arm64.

Furthermore, newer versions of Docker will issue the following warning:

WARN: FromPlatformFlagConstDisallowed: FROM --platform flag should not use constant value "linux/amd64"

NG 2: Addressing with docker build --no-cache

# Even if you clear the cache, the manifest format remains the same
docker build --no-cache -t <image> .

--no-cache rebuilds the image from scratch without using the build cache, but the format of the generated manifest (OCI Image Index) does not change. It is not a fundamental solution.

NG 3: Addressing with docker rmi + Re-push

# Deleting the local image does not change the manifest on ECR
docker rmi <image>:latest
docker build -t <image>:latest .
docker push <image>:latest

Even if you delete the local image and rebuild it, the manifest format generated by docker build remains the same. Also, if layers remain on the ECR side, the manifest might not be updated due to Layer already exists.

Why these are insufficient (Summary)


Conclusion: Future Best Practices

# ECR Login
aws ecr get-login-password --region <region> | \
  docker login --username AWS --password-stdin <account-id>.dkr.ecr.<region>.amazonaws.com

# Build & Push (single command)
docker buildx build \
  --platform linux/amd64 \
  --provenance=false \
  -t <ECR-repository-URL>:latest \
  --push \
  ./app

Handling in CI/CD (GitHub Actions)

CI/CD environments are usually executed on x86_64 runners, so this problem is less likely to occur. However, if you are using ARM64 runners or performing multi-architecture builds, specify --platform and --provenance=false in the same way.

# GitHub Actions example
- name: Build and Push
  run: |
    docker buildx build \
      --platform linux/amd64 \
      --provenance=false \
      -t ${{ env.ECR_REGISTRY }}/${{ env.ECR_REPOSITORY }}:latest \
      --push \
      ./app

Summary

Item Content
Cause of Error Docker Desktop on Apple Silicon Mac generates and pushes images in the OCI Image Index (arm64) format.
Why it's hard to notice docker inspect shows amd64 locally, but the manifest on ECR remains as arm64.
Solution Command docker buildx build --platform linux/amd64 --provenance=false --push
--platform Specifies the build target as amd64.
--provenance=false Disables OCI Image Index format and uses Docker V2 Manifest format.
--push Pushes directly from buildx to the registry, preventing manifest inconsistencies.
ECR Precautions If old images remain, they may not be updated due to layer caching. Delete them before re-pushing.
ECS Precautions Force a pull of the new image using aws ecs update-service --force-new-deployment.

Discussion