Screenshot Testing with @Preview Composables

The implementation details can be found in this PR.
https://github.com/DroidKaigi/conference-app-2022/pull/97

The original idea came from here:
https://github.com/android/nowinandroid/pull/101

Unlike the original, I configured it to run tests without placing comparison images in the repository.

• Do not store images used for testing in the repository
• If a diff is found during testing, store it in the artifacts
• Comment the link to the diff on the PR (couldn't achieve this)

These were the specifications I worked with.

Kotlin-related

The work mainly involves setting up Showkase and Paparazzi.

Showkase Configuration

Configure KSP and Showkase in the build.gradle of the module containing the Composable with @Preview.
https://github.com/DroidKaigi/conference-app-2022/blob/19bef1610b1daeb42e58b7e88a0dd408d3fdf0d0/core-ui/build.gradle.kts

Creating a Test Module

Build.gradle

I prepared a module named preview-screenshots.

This is where Paparazzi runs.
Configure Paparazzi in its build.gradle.
https://github.com/DroidKaigi/conference-app-2022/blob/19bef1610b1daeb42e58b7e88a0dd408d3fdf0d0/preview-screenshots/build.gradle.kts

Kotlin

Basically, I copied and pasted everything from the reference PR.
https://github.com/DroidKaigi/conference-app-2022/blob/19bef1610b1daeb42e58b7e88a0dd408d3fdf0d0/preview-screenshots/src/test/java/io/github/droidkaigi/confsched2022/ScreenshotShowkaseModule.kt

https://github.com/DroidKaigi/conference-app-2022/blob/19bef1610b1daeb42e58b7e88a0dd408d3fdf0d0/preview-screenshots/src/test/java/io/github/droidkaigi/confsched2022/PreviewScreenshotTests.kt

Traps Encountered

Don't use Zulu

Paparazzi doesn't work if the JDK is Zulu, so I resolved this by using Temurin instead.
https://github.com/cashapp/paparazzi/issues/565

Don't use compileSdk 33

Since Paparazzi doesn't work with compileSdk 33, I addressed this by referring to the following comment.
https://github.com/cashapp/paparazzi/issues/489#issuecomment-1195674603

@get:Rule
val paparazzi = Paparazzi(
    environment = detectEnvironment().copy(
        platformDir = "${androidHome()}/platforms/android-31",
        compileSdkVersion = 31
    ),
    maxPercentDifference = 0.0,
)

*Note: Although the comment suggests setting compileSdkVersion to 32, it doesn't work, so you need to set it to 31.

By following these steps, you can capture screenshots of Composables with @Preview and perform screenshot testing.
./gradlew recordPaparazziDebug
./gradlew preview-screenshots:verifyPaparazziDebug

CI-related

Running Screenshot Tests

https://github.com/DroidKaigi/conference-app-2022/blob/19bef1610b1daeb42e58b7e88a0dd408d3fdf0d0/.github/workflows/ScreenShotTest.yml

      - name: Download Screenshots
        id: download-artifact
        uses: dawidd6/action-download-artifact@v2
        with:
          workflow: UpdateScreenshots.yml
          name: screenshots
          path: preview-screenshots/src/test/snapshots/images

Since the requirement this time is not to store comparison images in the repository, we download the comparison images from the artifacts. For this requirement, we needed to download artifacts without knowing exactly when they were created, so we used dawidd6/action-download-artifact.

      - name: Run screenshot tests
        id: verifyPaparazziDebug
        continue-on-error: true
        run: ./gradlew preview-screenshots:verifyPaparazziDebug

This is where the screenshot test is executed.

      - name: Error Screenshot tests
        continue-on-error: true
        if: ${{ hashFiles('preview-screenshots/out/failures/*.png') != '' }}
        uses: actions/upload-artifact@v3
        with:
          name: scrennshot-test-results
          path: preview-screenshots/out/failures

If there is a diff in the screenshot test results, it is uploaded to the artifacts.

      - name: Comment PR
        continue-on-error: true
        if: ${{ hashFiles('preview-screenshots/out/failures/*.png') != '' }}
        uses: thollander/actions-comment-pull-request@v1
        with:
          message: |
            There are differences in Compose previews. Please check your build and download the diff artifact.
            https://github.com/DroidKaigi/conference-app-2022/actions/workflows/ScreenShotTest.yml
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

When a diff is found during the screenshot test, a comment is posted to the PR.
*Note: Ideally, I wanted to include a link to the diff artifact, but I had to give up on that.

Updating Comparison Screenshots

https://github.com/DroidKaigi/conference-app-2022/blob/19bef1610b1daeb42e58b7e88a0dd408d3fdf0d0/.github/workflows/UpdateScreenshots.yml
Nothing particularly unusual is happening here. It records screenshots and uploads them to the artifacts when a PR is merged.

Finally

Paparazzi often became unstable in new environments, but I believe I've navigated all the pitfalls I could find. Please give it a try!!