iTranslated by AI
Playwright + GitHub Actions: Insights on E2E and VRT Environment Setup and CI/CD Integration
Introduction
Since I was in charge of Playwright environment setup and CI/CD integration for my work, I had been constantly thinking about the best configuration for E2E testing and VRT.
I have managed to refine it into a form that I am reasonably satisfied with, so I will document that knowledge here.
🎭 Playwright
A testing tool developed by Microsoft. It performs E2E testing with features like multi-browser support, auto-waiting, and parallel processing. It can also detect visual changes through screenshot comparison tests (VRT: Visual Regression Testing). It seems to be developed by the team that originally created Puppeteer.
Cypress is a common comparison tool, but recently, Playwright's weekly downloads seem to have surpassed it.
(Others include Selenium and paid E2E testing tools provided by vendors.)
Also, while Playwright officially supports Java, Python, and C#, we will be using TypeScript this time.
Requirements
First, within this repository, there are two packages: server (Express.js) and web (React.js), and the development environment is started using docker compose up. The DB is also defined in docker-compose.yml.
- E2E tests pass on GitHub Actions
- VRT passes on GitHub Actions
- Be able to reproduce the same results in both local and CI environments
In this article, E2E and VRT are defined as follows:
- E2E (End-to-End Testing): Testing to confirm that functions work correctly throughout the entire system.
- VRT (Visual Regression Testing): Testing to detect visual changes.
Source
Here is the source code for this project.
This is the GitHub Actions execution history for this repository. E2E tests and VRT are executed daily at 9:00 AM.
Overall Configuration
/playwright-e2e-stack
├── /.github/workflows
│ ├── e2e.yml
│ └── vrt.yml
├── /e2e
│ ├── /tests # E2E test cases
│ ├── Dockerfile # for E2E testing environment
│ └── playwright.config.ts # E2E testing settings
├── /server
│ └── Dockerfile
├── /web
│ ├── /web/tests/visual/mock/db.json # mock data for json-server
│ ├── /tests # VRT test cases
│ ├── Dockerfile
│ ├── Dockerfile.test # for VRT execution environment
│ ├── playwright.config.ts # VRT settings
│ └── docker-compose.vrt.yml
├── docker-compose.yml
└── docker-compose.e2e.yml
Thinking behind the directory structure for E2E and VRT
-
E2E Testing:
Since these are tests performed across the entire system, thee2edirectory is created at the same level asserverandweb. This makes it:- Clear that it is a system-wide test by placing it in the root directory.
- Easier to design tests considering both the
serverandweb.
-
VRT:
These tests detect visual changes in the UI and are specialized for the frontend. Therefore:- Placing test definitions under
webmakes the relationship with UI changes clear. - Easier for frontend developers to manage directly.
- Using API mocks allows testing without depending on the backend.
- Placing test definitions under
E2E (End-to-End Testing) Details
Key Points
Execute E2E tests with docker compose -f docker-compose.e2e.yml up.
services:
db:
image: postgres:latest
environment:
POSTGRES_USER: admin
POSTGRES_PASSWORD: password
POSTGRES_DB: mydb
ports:
- "5432:5432"
healthcheck:
test: pg_isready -U admin -d mydb
interval: 3s
timeout: 3s
retries: 5
server:
build:
context: ./server
dockerfile: Dockerfile
environment:
- DATABASE_URL=postgresql://admin:password@db:5432/mydb?schema=public
- WEB_ORIGIN=http://web
ports:
- "3000:3000"
command: >
sh -c "npx prisma migrate deploy && npm start"
depends_on:
db:
condition: service_healthy
web:
build:
context: ./web
dockerfile: Dockerfile
args:
- VITE_API_URL=http://server:3000
ports:
- "80:80"
e2e:
build:
context: ./
dockerfile: ./e2e/Dockerfile
environment:
- CI=${CI}
- DATABASE_URL=postgresql://admin:password@db:5432/mydb?schema=public
- PLAYWRIGHT_BASE_URL=http://web
- PLAYWRIGHT_API_URL=http://server:3000
volumes:
- ./e2e/test-results:/app/test-results
ports:
- "9323:9323"
depends_on:
- web
The process follows this sequence:
-
Starting the DB container
- Starts the Postgres database.
- Runs a health check to confirm that the database is ready.
-
Starting the Server container
- Waits for the DB container's health check to succeed.
- Builds the server application.
- Executes Prisma migrations.
- Starts the server application.
-
Starting the Web container
- Builds the frontend application.
- Starts the web server.
-
Starting the E2E Test container
- Builds the E2E testing environment.
- Waits for the Web container to start.
- Prepares for test execution.
-
Executing tests
- Executes Playwright tests inside the E2E test container.
- Saves the test results to the directory mounted as a volume.
FROM mcr.microsoft.com/playwright:latest
EXPOSE 9323
WORKDIR /app
# By running npm ci before copying the source code, we can cache the dependency layer
COPY ./e2e/package*.json .
RUN npm ci
COPY ./e2e .
COPY ./server/prisma ./prisma
RUN npx prisma generate
CMD ["npm", "test"]
For the test execution environment, we use Playwright's official "mcr.microsoft.com/playwright" as the base image. Adopting the official image allows us to skip the manual setup of dependencies required for test execution, significantly reducing the effort needed for environment construction.
Furthermore, by using a containerized testing environment, we can minimize the differences in the execution environment between local development and CI. This prevents inconsistencies in test results caused by environmental differences, leading to more reliable and consistent outcomes.
name: End-to-End Testing
run-name: E2E Testing By @${{ github.actor }}
on:
workflow_dispatch:
push:
branches: ["main"]
schedule:
# Daily at 9:00 (JST)
- cron: "0 0 * * *"
jobs:
e2e:
timeout-minutes: 10
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: docker compose build
run: docker compose -f docker-compose.e2e.yml build
- name: Run Playwright tests
run: docker compose -f docker-compose.e2e.yml up --abort-on-container-exit --exit-code-from e2e
- uses: actions/upload-artifact@v4
if: always()
with:
name: playwright-report
path: e2e/playwright-report/
retention-days: 30
- name: Cleanup Docker resources
if: always()
run: docker compose -f docker-compose.e2e.yml down --rmi all --volumes --remove-orphans
Here is an explanation of the GitHub Actions workflow:
-
Trigger Settings:
For convenience, it's set to run on pushes to the main branch, but E2E tests often take more time than other tests. Therefore, it is generally recommended to run them on a schedule (daily) for specific branches. -
Executing Tests:
Tests are executed viadocker compose -f docker-compose.e2e.yml up --abort-on-container-exit --exit-code-from e2e(same as in the local environment). The--abort-on-container-exitflag ensures that when any container exits, all other containers are stopped, and--exit-code-from e2euses the exit code of the e2e container as the overall command's exit code. -
Uploading Test Reports:
The test reports created undere2e/playwright-reportare uploaded as artifacts. This allows you to check the details of tests performed on GitHub Actions. -
Cleaning Up Resources:
docker compose down --rmi all --volumes --remove-orphansdeletes related resources. Not leaving unnecessary data or settings helps mitigate potential security risks.
I borrowed the command from here.
GitHub Actions Test Report
After running the tests on GitHub Actions, downloading the generated "playwright-report" and running npx playwright show-report [downloaded-path] allows you to check detailed test results and failure screenshots, helping you identify and solve issues.
npx playwright show-report ~/Downloads/
# or
npx playwright show-report ~/Downloads/playwright-report etc.
Also, instead of downloading the test results, it seems they can be hosted on GitHub Pages. That's a nice touch.
VRT (Visual Regression Testing) Details
Where to store the snapshots?
First, concerning where to store the captures used for image comparison, the following candidates can be considered:
- Cloud storage such as S3
- GitHub Actions Artifacts
- Capturing a baseline for comparison during every execution
- Including them in the commit
Reference
I decided to include them in the commits because I wanted to use a format that is as independent of external factors as possible.
API Mock Server
For the API mock server, I am using json-server.
{
"users": [
{
"id": 1,
"name": "John Doe",
"email": "john.doe@example.com"
},
{
"id": 2,
"name": "Jane Smith",
"email": "jane.smith@example.com"
}
]
}
By preparing a JSON file like the one above, you can easily set up a convenient REST API.
Other Key Points
Similar to the E2E tests, the tests are executed with docker compose -f ./web/docker-compose.vrt.yml up.
services:
vrt:
build:
context: ./
dockerfile: Dockerfile.test
args:
- VITE_API_URL=http://localhost:3000
environment:
- CI=${CI}
ports:
- "80:80"
- "3000:3000"
- "9323:9323"
volumes:
- ./playwright-report:/app/playwright-report
- ./test-results:/app/test-results
- ./snapshots:/app/snapshots
- ./tests/:/app/tests
For VRT, only the web directory is used.
To reiterate, the focus of VRT is the frontend. (Although the philosophy might differ by project...) Therefore, I chose a configuration that is entirely self-contained within the web directory.
FROM mcr.microsoft.com/playwright:latest
EXPOSE 80
WORKDIR /app
COPY package*.json .
RUN npm ci
ARG VITE_API_URL
ENV VITE_API_URL=$VITE_API_URL
COPY . .
RUN npm run build
CMD ["npm", "test"]
I am using the official Playwright image for the same reasons as in E2E.
Especially in VRT, device differences between snapshots taken on (my local) Mac and those taken on GitHub Actions become very apparent. To avoid this, a containerized testing environment seems essential.
webServer: [
{
command: "npm start",
url: 'http://localhost',
reuseExistingServer: !process.env.CI,
},
{
command: "npm run mock",
url: 'http://localhost:3000',
reuseExistingServer: !process.env.CI,
},
],
While in E2E, server startups were entirely managed by docker-compose.yml, in VRT, I use the webServer field in playwright.config.ts to launch the necessary servers when the tests are executed.
name: Visual Regression Testing
run-name: VRT By @${{ github.actor }}
on:
workflow_dispatch:
push:
branches: ["main"]
schedule:
# Daily at 9:00 (JST)
- cron: "0 0 * * *"
jobs:
vrt:
timeout-minutes: 10
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: docker compose build
run: docker compose -f ./web/docker-compose.vrt.yml build
- name: Run Playwright tests
run: docker compose -f ./web/docker-compose.vrt.yml up --abort-on-container-exit --exit-code-from vrt
- uses: actions/upload-artifact@v4
if: always()
with:
name: playwright-report
path: web/playwright-report/
retention-days: 30
- name: Cleanup Docker resources
if: always()
run: docker compose -f ./web/docker-compose.vrt.yml down --rmi all --volumes --remove-orphans
The workflow is identical to E2E except for the directory and docker-compose.yml specifications.
Summary
In this article, I explained the implementation of E2E testing and VRT using Playwright. Here are the key points:
-
Environment Construction:
- Built a consistent testing environment for both local and CI environments using
docker-compose. - Minimized environmental dependencies by using the official Playwright image.
- Built a consistent testing environment for both local and CI environments using
-
E2E Testing:
- Targeted the entire system and achieved testing integrated with the actual backend.
- Placed the
e2edirectory at the root to clarify that these are system-wide tests.
-
VRT (Visual Regression Testing):
- Specialized for the frontend to detect visual changes in the UI.
- Enabled frontend-only testing by using a mock server.
-
CI/CD Integration:
- Realized automatic test execution and report generation using GitHub Actions.
- Achieved continuous quality assurance by running tests daily against specific branches.
First, regarding VRT: initially, I tried to perform VRT and E2E tests on the same scope, and I struggled a lot with sharing environments. After reconsidering and organizing the concerns of each, I believe I've arrived at a reasonably simple configuration. (This was the second point where I stumbled.)
Also, I realized I had been using many parts of docker-compose "somehow" without fully understanding them, so I took this opportunity to relearn. Feel free to take a look if you'd like. (laughs)
E2E testing is easy to introduce and cost-effective as it allows verification of the entire system. However, it also comes with high maintenance costs. Starting with E2E tests, I want to ensure that integration and unit tests are also thoroughly implemented without neglect!
Finally, a big thank you to everyone who gave me various pieces of advice!
Appendix
Obstacle 1
Initially, I considered a configuration where docker-compose only managed the server and db, while the web was started in the local environment (host machine).
In this case, communication from web to server works via localhost in the local environment, but it fails on GitHub Actions.
As described in the article, managing the web with docker-compose as well allows them to communicate with each other through the Docker Network.
Obstacle 2
Initially, I viewed VRT as a type of E2E and managed both in the e2e directory.
The shared docker-compose.yml and playwright.config.ts resulted in an implementation with very low readability and maintainability.
This time, I completely separated them by defining E2E under the e2e directory and VRT under the web directory.
There may be cases where VRT is performed End-to-End. In such cases, I recommend building separate environments for each rather than forcing commonization.
Obstacle 3
Depending on the project, there were cases where service startup order could not be fully controlled even when using depends_on or healthcheck in docker-compose. In such cases, you can control it by using wait-for-it or dockerize.
This is also introduced in the official documentation.
Discussion