テスト基盤としてのStorybook7

概要

• Test runner to automatically test your entire Storybook and catch broken stories.
• Visual tests capture a screenshot of every story then compare it against baselines to detect appearance and integration issues
• Accessibility tests catch usability issues related to visual, hearing, mobility, cognitive, speech, or neurological disabilities
• Interaction tests verify component functionality by simulating user behaviour, firing events, and ensuring that state is updated as expected
• Coverage tests to measure how much of your code is covered by your tests
• Snapshot tests detect changes in the rendered markup to surface rendering errors or warnings
• Import stories in other tests to QA even more UI characteristics

These tests run in a live browser and can be executed via the command line or your CI server.

テストのためにブラウザを立ち上げる必要がある

The test-runner is powered by Jest and accepts a subset of its CLI options (for example, --watch, --maxWorkers). If you're already using any of those flags in your project, you should be able to migrate them into Storybook's test-runner without any issues. Listed below are all the available flags and examples of using them.

JestのCLIオプションみたいなので細々設定できる

By default, the test-runner assumes that you're running it against a locally served Storybook on port 6006. If you want to define a target URL to run against deployed Storybooks, you can use the --url flag or set the TARGET_URL environment variable. For example:

deploy済みのStorybookに対してもテストを実行できる

なのでたぶんCommitごとにStorybookをdeployしてそれに対してCI回すみたいなこともできる

# .github/workflows/storybook-tests.yml

name: Storybook Tests
on: deployment_status
jobs:
  test:
    timeout-minutes: 60
    runs-on: ubuntu-latest
    if: github.event.deployment_status.state == 'success'
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-node@v3
        with:
          node-version: '16.x'
      - name: Install dependencies
        run: yarn
      - name: Install Playwright
        run: npx playwright install --with-deps
      - name: Run Storybook tests
        run: yarn test-storybook
        env:
          TARGET_URL: '${{ github.event.deployment_status.target_url }}'

今まではChromaticでやってたけど、これからは併用かもね

• Use it locally and Chromatic on your CI.
• Use Chromatic for visual and interaction tests and run other custom tests using the test runner.

escape hatch的な位置づけでtest hookを用意している

// .storybook/test-runner.ts

import type { TestRunnerConfig } from '@storybook/test-runner';

const config: TestRunnerConfig = {
  // Hook that is executed before the test runner starts running tests
  setup() {
    // Add your configuration here.
  },
  /* Hook to execute before a story is rendered.
   * The page argument is the Playwright's page object for the story.
   * The context argument is a Storybook object containing the story's id, title, and name.
   */
  async preRender(page, context) {
    // Add your configuration here.
  },
  /* Hook to execute after a story is rendered.
   * The page argument is the Playwright's page object for the story
   * The context argument is a Storybook object containing the story's id, title, and name.
   */
  async postRender(page, context) {
    // Add your configuration here.
  },
};

module.exports = config;

カタログとしてのStorybookはデフォルトで書くとして、必要な箇所にplay functionを書いていって、それがCIされていれば最高ですね

定義

Visual tests, also called visual regression tests, catch bugs in UI appearance. They work by taking screenshots of every story and comparing them commit-to-commit to identify changes.

機能面のテストではキャッチできない見た目の差分を把握するためのテスト

以前、Tailwind の設定ファイルを触って思わぬマージンが発生した事例を見かけたことがあるのでそういうときに良さそう

基本的にはChromaticでやる

What’s the difference between visual tests and snapshot tests?
Snapshot tests compare the rendered markup of every story against known baselines. This means the test compares blobs of HTML and not what the user actually sees. Which in turn, can lead to an increase in false positives as code changes don’t always yield visual changes in the component.

visual testはスクショの比較
snapshot testはHTMLの塊の比較

HTMLが変更されてても見た目は変更されてないことはよくある
逆はあるかな？

画像の差し替えとかはあるか。pathは同じだけど中身変わってます〜みたいな。
他にもたくさんあるんだろうな。

定義

Accessibility is the practice of making websites inclusive to all. That means supporting requirements such as: keyboard navigation, screen reader support, touch-friendly, usable color contrast, reduced motion, and zoom support.

Accessibility tests audit the rendered DOM against a set of heuristics based on WCAG rules and other industry-accepted best practices. They act as the first line of QA to catch blatant accessibility violations.

ChromeのDeveloper toolsにもあるよねこの手のチェック

Storybook provides an official a11y addon. Powered by Deque's axe-core, which automatically catches up to 57% of WCAG issues.

57%がどれほどの数字なのかわからない...
基本的なところは確実にチェックしているっていう感じかな？

一気にたくさん問題出てきたらストレス大きすぎて無視しちゃいそう

The most accurate way to check accessibility is manually on real devices. However, you can use automated tools to catch common accessibility issues. For example, Axe, on average, catches upwards to 57% of WCAG issues automatically.

These tools work by auditing the rendered DOM against heuristics based on WCAG rules and other industry-accepted best practices. You can then integrate these tools into your test automation pipeline using the Storybook test runner and axe-playwright.

これもCI

a11y的にbrokenでも「あとで直しましょう！」みたいな意思決定が容易に起きちゃう気がするな
それでも可視化しておくのは大事だけれども

What’s the difference between browser-based and linter-based accessibility tests?
Browser-based accessibility tests, like those found in Storybook, evaluate the rendered DOM because that gives you the highest accuracy. Auditing code that hasn't been compiled yet is one step removed from the real thing, so you won't catch everything the user might experience.

たしかに、Storybookでa11yを見る意義は大きそう

定義

As you build more complex UIs like pages, components become responsible for more than just rendering the UI. They fetch data and manage state. Interaction tests allow you to verify these functional aspects of UIs.

Since Storybook is a webapp, anyone with the URL can reproduce the error with the same detailed information without any additional environment configuration or tooling required.

これいいよなー

What's the difference between interaction tests and using Jest + Testing Library alone?
Interaction tests integrate Jest and Testing Library into Storybook. The biggest benefit is the ability to view the component you're testing in a real browser. That helps you debug visually, instead of getting a dump of the (fake) DOM in the command line or hitting the limitations of how JSDOM mocks browser functionality. It's also more convenient to keep stories and tests together in one file than having them spread across files.

storyファイルにテストがすべて集約されるというのが、本スクラップのタイトルにも関わるところ

What’s the difference between interaction tests and visual tests?
Interaction tests can be expensive to maintain when applied wholesale to every component. We recommend combining them with other methods like visual testing for comprehensive coverage with less maintenance work.

効果的にカバレッジを生み出すための使い分けかー
これはやってみてベストプラクティスを生み出す系のやつだ

定義

Test coverage is the practice of measuring whether existing tests fully cover your code. That means surfacing areas which aren't currently being tested, such as: conditions, logic branches, functions and variables.

どういう観点でテストを設計すべきなのか全くわかってないので知りたい

test coverageの計測はビルドツールごとに設定が必要
レシピがこのリポジトリに載っている
https://github.com/yannbf/storybook-coverage-recipes

testといいつつ、失敗しても失敗とはかぎらないんだな（？）

Snapshotの保存は誰がやってくれるんだ？Storyshot？

そして、snapshots/Storyshots.test.ts.snapというファイルが作成されました。

こういう感じで生成されるらしい
https://kaminashi-developer.hatenablog.jp/entry/expo-storyshots

いちいち保存するのめんどいな

Stories are a practical starting point for UI testing. Import stories into tools like Jest, Testing Library, Puppeteer, Cypress, and Playwright to save time and maintenance work.

テスト基盤としてのStorybookだ

やっぱplay functionの中でE2Eみたいなこと書けるの強い