iTranslated by AI

The content below is an AI-generated translation. This is an experimental feature, and may contain errors. View original article
🖼️

Released a GitHub Custom Action to Simplify Visual Regression Testing (VRT) Implementation

に公開
Flutter
GitHub Actions
VRT
tech

I have released a GitHub Actions custom action called Easy VRT. It consists of an action and workflow templates that allow you to easily create Visual Regression Test (VRT) workflows.

https://github.com/yorifuji/easy-vrt

In VRT, it is necessary to incorporate two tasks into the CI workflow: image capturing and regression testing.

Image capturing requires two types of images: "expected" images, which treat the merge target branch (master in the figure below) as the correct result, and "actual" images from the topic branch. It is necessary to identify the commit to be captured based on the Git history.


From the reg-suit website

In regression testing, there is a technique to shorten the overall processing time by using images captured in the past (especially expected images) as a cache and reusing them from the second time onwards. It is common to use external storage (S3/Google Cloud Storage, etc.) as a storage location for caches and test result reports, which requires cloud service settings and preparation of credentials.

Since these processes must be carried out within the CI, VRT workflows and settings tend to become complex.

Custom Action

The published custom action provides general-purpose functionality and workflow templates for VRT.

The features are as follows:

  • The custom action performs common processes required for VRT
  • Template workflows provide a CI boilerplate
  • Completed entirely within GitHub Actions without using external storage
  • Performs regression testing using reg-cli
  • Regression test results are uploaded to Artifacts; although it involves the extra step of downloading, we have decided to accept this

An actual execution example can be seen in this repository.

PR
https://github.com/yorifuji/easy-vrt-example/pull/1

Workflow
https://github.com/yorifuji/easy-vrt-example/actions/runs/7237123682

About the Use Case

This action is based on what I used in Flutter app development. Flutter supports Golden File Testing (VRT) at the framework level, making it a platform where it is easy to take screenshots of the app.

On the other hand, in environments such as languages or platforms where taking screenshots requires deploying artifacts or actually accessing a server, and where the workflow for obtaining screenshots inevitably becomes complex, it might be difficult to use this action.

Template Workflow

The template workflow consists of the following jobs:

While fine-tuning is required for each environment, the key point is to add steps for creating images in the two locations described later.

Entire template workflow

name: vrt

run-name: visual regression test

on: pull_request

permissions:
  contents: read

jobs:
  lookup:
    runs-on: ubuntu-latest
    outputs:
      actual-sha: ${{ steps.lookup.outputs.actual-sha }}
      actual-cache-hit: ${{ steps.lookup.outputs.actual-cache-hit }}
      expected-sha: ${{ steps.lookup.outputs.expected-sha }}
      expected-cache-hit: ${{ steps.lookup.outputs.expected-cache-hit }}
    steps:
      - uses: yorifuji/easy-vrt@v1
        id: lookup
        with:
          mode: lookup

  expected:
    if: ${{ !cancelled() && !failure() && needs.lookup.outputs.expected-cache-hit != 'true' }}
    needs: lookup
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          ref: ${{ needs.lookup.outputs.expected-sha }}

      # >>> add step to create expected image

      # <<< add step to create expected image

      - uses: yorifuji/easy-vrt@v1
        with:
          mode: expected
          expected-dir: your-expected-image-dir # set the directory where the expected image is stored
          expected-cache-key: ${{ needs.lookup.outputs.expected-sha }}

  actual:
    if: ${{ !cancelled() && !failure() && needs.lookup.outputs.actual-cache-hit != 'true' }}
    needs: lookup
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          ref: ${{ needs.lookup.outputs.actual-sha }}

      # >>> add step to create actual image

      # <<< add step to create actual image

      - uses: yorifuji/easy-vrt@v1
        with:
          mode: actual
          actual-dir: your-actual-image-dir # set the directory where the actual image is stored
          actual-cache-key: ${{ needs.lookup.outputs.actual-sha }}

  compare:
    if: ${{ !cancelled() && !failure() }}
    needs: [lookup, expected, actual]
    runs-on: ubuntu-latest
    steps:
      - uses: yorifuji/easy-vrt@v1
        with:
          mode: compare
          expected-cache-key: ${{ needs.lookup.outputs.expected-sha }}
          actual-cache-key: ${{ needs.lookup.outputs.actual-sha }}

Copy the above workflow as is and describe the steps to generate images in the following two locations of the expected and actual jobs. Pass the path to the folder where the images are saved to the action's parameters.

      # >>> add step to create expected image

      # <<< add step to create expected image

      - uses: yorifuji/easy-vrt@v1
        with:
          mode: expected
          expected-dir: your-expected-image-dir # set the directory where the expected image is stored
          expected-cache-key: ${{ needs.lookup.outputs.expected-sha }}


      # >>> add step to create actual image

      # <<< add step to create actual image

      - uses: yorifuji/easy-vrt@v1
        with:
          mode: actual
          actual-dir: your-actual-image-dir # set the directory where the actual image is stored
          actual-cache-key: ${{ needs.lookup.outputs.actual-sha }}

Using a Flutter app as an example, the code for generating the expected image would look like this. Describe the same process for actual as well. It might be a good idea to share the generation process using Reusable workflows.

      - uses: subosito/flutter-action@v2

      - run: |
	  flutter pub get
	  flutter test --update-goldens --tags=golden

      - uses: yorifuji/easy-vrt@main
        with:
          mode: expected
          expected-dir: test/golden_test/goldens
          expected-cache-key: ${{ needs.lookup.outputs.expected-sha }}

      - uses: subosito/flutter-action@v2

      - run: |
	  flutter pub get
	  flutter test --update-goldens --tags=golden

      - uses: yorifuji/easy-vrt@main
        with:
          mode: actual
          actual-dir: test/golden_test/goldens
          actual-cache-key: ${{ needs.lookup.outputs.actual-sha }}

Once the actual and expected jobs are finished, tests are conducted in the compare job. Internally, it uses reg-cli. The result reports are uploaded to Artifacts.

When you download and extract it, you will find the report generated by reg-cli.

Options are also available to write comments to the Job Summary and Pull Requests.


Content displayed in the workflow summary


Content displayed in Pull Request comments

Job Details

About the jobs in the template workflow.

lookup

lookup
  lookup:
    runs-on: ubuntu-latest
    outputs:
      actual-sha: ${{ steps.lookup.outputs.actual-sha }}
      actual-cache-hit: ${{ steps.lookup.outputs.actual-cache-hit }}
      expected-sha: ${{ steps.lookup.outputs.expected-sha }}
      expected-cache-hit: ${{ steps.lookup.outputs.expected-cache-hit }}
    steps:
      - uses: yorifuji/easy-vrt@v1
        id: lookup
        with:
          mode: lookup

This job checks the SHAs targeted for image generation from the Pull Request's HEAD and base branches and confirms the existence of a cache. This job is required when using the template workflow, so there is no need to edit it.

expected, actual

expected, actual
  expected:
    if: ${{ !cancelled() && !failure() && needs.lookup.outputs.expected-cache-hit != 'true' }}
    needs: lookup
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          ref: ${{ needs.lookup.outputs.expected-sha }}

      # >>> add step to create expected image

      # <<< add step to create expected image

      - uses: yorifuji/easy-vrt@v1
        with:
          mode: expected
          expected-dir: your-expected-image-dir # set the directory where the expected image is stored
          expected-cache-key: ${{ needs.lookup.outputs.expected-sha }}

  actual:
    if: ${{ !cancelled() && !failure() && needs.lookup.outputs.actual-cache-hit != 'true' }}
    needs: lookup
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          ref: ${{ needs.lookup.outputs.actual-sha }}

      # >>> add step to create actual image

      # <<< add step to create actual image

      - uses: yorifuji/easy-vrt@v1
        with:
          mode: actual
          actual-dir: your-actual-image-dir # set the directory where the actual image is stored
          actual-cache-key: ${{ needs.lookup.outputs.actual-sha }}

These jobs generate the expected and actual images based on the hashes obtained in the lookup step. If a cache is found during the lookup job, these jobs will be skipped.

When using the template workflow in your environment, add steps to generate images to these jobs. Set the directories where the generated images are saved to expected-dir and actual-dir.

compare

compare
  compare:
    if: ${{ !cancelled() && !failure() }}
    needs: [lookup, expected, actual]
    runs-on: ubuntu-latest
    steps:
      - uses: yorifuji/easy-vrt@v1
        with:
          mode: compare
          expected-cache-key: ${{ needs.lookup.outputs.expected-sha }}
          actual-cache-key: ${{ needs.lookup.outputs.actual-sha }}

This job performs regression testing on the expected and actual images using reg-cli. The resulting report is uploaded to Artifacts. If you want to use the optional features for posting comments to the Job Summary or Pull Request, set the appropriate flags (refer to the README for details).

Custom Action Implementation

This is the internal implementation of the custom action (yorifuji/easy-vrt) used in the workflow; you don't need to know the contents if you just want to use the template workflow. Please refer to this only if you are interested.

lookup

Identifies the Git SHAs corresponding to actual and expected. actual uses the HEAD of the topic branch, while expected uses the common parent of the base branch and the topic branch (git merge-base).

Once the SHAs are determined, it checks for the existence of a cache using the lookup-only: flag of actions/cache.

The hash values and the presence/absence of cache are used in subsequent jobs via outputs.

lookup
    # lookup

    - if: ${{ inputs.mode == 'lookup' }}
      uses: actions/checkout@v4
      with:
        fetch-depth: 0

    - if: ${{ inputs.mode == 'lookup' }}
      id: lookup-sha
      shell: bash
      run: |
        echo "actual-sha=${{ github.event.pull_request.head.sha }}" >> $GITHUB_OUTPUT
        echo "expected-sha=$(git merge-base ${{ github.event.pull_request.base.sha }} ${{ github.event.pull_request.head.sha }})" >> $GITHUB_OUTPUT

    - if: ${{ inputs.mode == 'lookup' }}
      id: lookup-actual-cache
      uses: actions/cache@v3
      with:
        key: reg-suit-cache-${{ steps.lookup-sha.outputs.actual-sha }}
        path: .easy-vrt/actual
        lookup-only: true

    - if: ${{ inputs.mode == 'lookup' }}
      id: lookup-expected-cache
      uses: actions/cache@v3
      with:
        key: reg-suit-cache-${{ steps.lookup-sha.outputs.expected-sha }}
        path: .easy-vrt/expected
        lookup-only: true

expected, actual

Performs the process of saving the images received via parameters to the cache using actions/cache/save.

actual, expected
    # expected

    - if: ${{ inputs.mode == 'expected' }}
      shell: bash
      run: |
        if [ -e .easy-vrt ]; then exit 1; fi
        mkdir .easy-vrt && mv ${{ inputs.expected-dir }} .easy-vrt/expected

    - if: ${{ inputs.mode == 'expected' }}
      uses: actions/cache/save@v3
      with:
        path: .easy-vrt/expected
        key: reg-suit-cache-${{ inputs.expected-cache-key }}

    # actual

    - if: ${{ inputs.mode == 'actual' }}
      shell: bash
      run: |
        if [ -e .easy-vrt ]; then exit 1; fi
        mkdir .easy-vrt && mv ${{ inputs.actual-dir }} .easy-vrt/actual

    - if: ${{ inputs.mode == 'actual' }}
      uses: actions/cache/save@v3
      with:
        path: .easy-vrt/actual
        key: reg-suit-cache-${{ inputs.actual-cache-key }}

compare

It is a bit long, but it performs the following processes:

  • Restores actual and expected data from the cache
  • Installs reg-cli via npm
  • Executes npx reg-cli
  • Uploads the report to Artifacts
  • (Optional) Outputs a summary to the Job Summary and Pull Request
compare
    # compare

    - if: ${{ inputs.mode == 'compare' }}
      shell: bash
      run: |
        echo expected-cache-key ${{ inputs.expected-cache-key }}
        echo actual-cache-key ${{ inputs.actual-cache-key }}

    - if: ${{ inputs.mode == 'compare' }}
      shell: bash
      run: |
        if [ -e .easy-vrt ]; then exit 1; fi

    - if: ${{ inputs.mode == 'compare' }}
      shell: bash
      run: |
        if [ -e package.json ]; then exit 1; fi
        if [ -e package-lock.json ]; then exit 1; fi
        cp $GITHUB_ACTION_PATH/package.json $GITHUB_ACTION_PATH/package-lock.json .

    - if: ${{ inputs.mode == 'compare' }}
      uses: actions/setup-node@v4
      with:
        node-version: 20
        cache: npm

    - if: ${{ inputs.mode == 'compare' }}
      shell: bash
      run: npm install

    - if: ${{ inputs.mode == 'compare' }}
      uses: actions/cache/restore@v3
      with:
        key: reg-suit-cache-${{ inputs.actual-cache-key }}
        path: .easy-vrt/actual

    - if: ${{ inputs.mode == 'compare' }}
      uses: actions/cache/restore@v3
      with:
        key: reg-suit-cache-${{ inputs.expected-cache-key }}
        path: .easy-vrt/expected

    - if: ${{ inputs.mode == 'compare' }}
      shell: bash
      run: >
        npx reg-cli
        .easy-vrt/actual
        .easy-vrt/expected
        .easy-vrt/diff
        -R .easy-vrt/report.html
        -J .easy-vrt/report.json

    - if: ${{ !cancelled() && inputs.mode == 'compare' }}
      uses: actions/upload-artifact@v3
      with:
        name: easy-vrt-report
        path: .easy-vrt
        # retention-days: 3

    - if: ${{ !cancelled() && inputs.mode == 'compare' }}
      uses: actions/github-script@v7
      env:
        SUMMARY_COMMENT: ${{ inputs.summary-comment }}
        REVIEW_COMMENT: ${{ inputs.review-comment }}
      with:
        script: |
          const fs = require('fs');

          const summaryComment = process.env.SUMMARY_COMMENT === 'true';
          const reviewComment = process.env.REVIEW_COMMENT === 'true';

          const log = fs.readFileSync('.easy-vrt/report.json', 'utf-8');
          console.log(log);
          const json = JSON.parse(log);
          console.log(json);

          const titleIcon = '✅';
          const easyVrtComment = '<!-- Easy VRT Comment -->';

          const stats = {
            changed: json.failedItems.length.toString(),
            newItems: json.newItems.length.toString(),
            deleted: json.deletedItems.length.toString(),
            passing: json.passedItems.length.toString()
          };

          const markdown = await core.summary
            .addHeading(`${titleIcon} easy-vrt has checked for visual changes`, 3)
            .addTable([
              ["🔴 Changed",  "🟡 New",        "🟤  Deleted",  "🔵 Passing"],
              [stats.changed, stats.newItems, stats.deleted, stats.passing]
            ])
            .addHeading("📊 Download Report", 3)
            .addLink('You can download the report from the artifact here', `https://github.com/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}`)
            .addRaw(`${easyVrtComment}`)
            .stringify()

          if (summaryComment) {
            await core.summary.write()
          }

          if (reviewComment) {
            const requestPerPage = 100;
            try {
              const listComments = await github.rest.issues.listComments({
                owner: context.repo.owner,
                repo: context.repo.repo,
                issue_number: context.issue.number,
                per_page: requestPerPage
              });
              const easyVrtCommentList = listComments.data.find(comment => comment.body.includes(easyVrtComment));
              if (easyVrtCommentList) {
                // delete summary comment if it exists
                await github.rest.issues.deleteComment({
                  owner: context.repo.owner,
                  repo: context.repo.repo,
                  comment_id: easyVrtCommentList.id
                });
              }
              // create a comment
              await github.rest.issues.createComment({
                owner: context.repo.owner,
                repo: context.repo.repo,
                issue_number: context.issue.number,
                body: markdown
              });
            } catch (error) {
              logError(`Failed to modify comment: ${error.message}`);
              return;
            }
          }

The key point is that it uses reg-cli instead of reg-suit for regression testing. reg-cli is a CLI tool that omits some features of reg-suit. Additionally, by using the -J, --json option, results can be output in JSON format. I have implemented a feature that parses the output JSON file to post comments to the Job Summary or Pull Requests.

Uploading to Artifacts and log analysis are implemented in JavaScript using actions/github-script.

About Image Generation

For Flutter, you can find articles by searching for "Flutter VRT," so please refer to those.

Summary

By templating the VRT workflow, I have made it easy to use.

Enjoy your VRT life!

Discussion