🤖

mix testでテストが実行される過程を追ってみる

2021/09/16に公開

Elixirで mix new からMix projectを作成すると

  • test/test_helper.exs
  • test/<project名>_test.exs

といったファイルが自動で作られており、何も意識せずとも mix test を実行すればテストがすでに実行できる状態になっています。

# test/test_helper.exs
ExUnit.start()
# test/sample_test.exs
defmodule SampleTest do
  use ExUnit.Case
  doctest Sample

  test "greets the world" do
    assert Sample.hello() == :world
  end
end
$ mix test
Compiling 1 file (.ex)
Generated sample app
..

Finished in 0.04 seconds (0.00s async, 0.04s sync)
1 doctest, 1 test, 0 failures

Randomized with seed 331689

これらが何をやっているのか、深堀りしてみたいと思います。

ExUnit

mix testを掘っていく前に、ExUnitについて抑えていきましょう。hexdocsの冒頭に

Unit testing framework for Elixir.

と記載されている通り、ExUnitはElixirのテスティングフレームワークです。

https://hexdocs.pm/ex_unit/1.12/ExUnit.html

最小構成でテストを実行してみる

テストの環境は mix newmix phx.new したらついてくるものという印象が強い(?)ですが、Elixir本体に組み込まれているものなので、実はmix projectでなくても実行可能です。以下、hexdocsの例をそのまま持ってきました。

# File: assertion_test.exs

# 1) Start ExUnit.
ExUnit.start()

# 2) Create a new test module (test case) and use "ExUnit.Case".
defmodule AssertionTest do
  # 3) Note that we pass "async: true", this runs the test case
  #    concurrently with other test cases. The individual tests
  #    within each test case are still run serially.
  use ExUnit.Case, async: true

  # 4) Use the "test" macro instead of "def" for clarity.
  test "the truth" do
    assert true
  end
end
$ elixir assertion_test.exs 
.

Finished in 0.05 seconds (0.05s on load, 0.00s async, 0.00s sync)
1 test, 0 failures

Randomized with seed 950675
  1. ExUnit.start/0 を実行
  2. use ExUnit.case を記述したモジュールを記述
  3. async: true をオプションとして渡せば並列に実行される
  4. test マクロを使ってテストを記述

というステップを踏めば、もうテストが実行できます。

ExUnit.Case

use ExUnit.Case

これの記述のおかげで test ...dodescribe ... do のようなマクロが使えるようになります。試しにuseの記述を消して実行するとCompileErrorになります。

$ elixir assertion_test.exs
** (CompileError) assertion_test.exs:12: undefined function test/2

内部では ExUnit.Callbacks でsetupやon_exitなどのcallbackが定義されていたり、 ExUnit.Assertions でassertやrefuteなどのmacroが定義されていたりします。これらをひっくるめて use ExUnit.Case さえ呼べばこれらのモジュール群が読み込めるという作りになっています。

# lib/ex_unit/lib/ex_unit/case.ex
  @doc false
  defmacro __using__(opts) do
    unless Process.whereis(ExUnit.Server) do
      raise "cannot use ExUnit.Case without starting the ExUnit application, " <>
              "please call ExUnit.start() or explicitly start the :ex_unit app"
    end

    quote do
      unless ExUnit.Case.__register__(__MODULE__, unquote(opts)) do
        use ExUnit.Callbacks
      end

      import ExUnit.Callbacks
      import ExUnit.Assertions
      import ExUnit.Case, only: [describe: 2, test: 1, test: 2, test: 3]
      import ExUnit.DocTest
    end
  end

mixとの統合

elixirでは mix test というMixタスクが標準で提供(コード参考)されており、 mix test を実行すると

  1. test/test_helper.exs を読み込み
  2. test/**/*_test.exs を並列に実行

という順序でテストケースが実行されます。これがこの記事冒頭に書いた2つのファイルであり、最初に読み込まれる test_helper.exsExUnit.start() が記載されているため、テストが実行できるというわけです。

「じゃExUnit.startは何をやっているの?」まではちょっと大変ですね。この記事のスコープからはいったん外してヨシということで。興味がある人はぜひ辿ってみて詳細教えてください。

https://github.com/elixir-lang/elixir//blob/4d318398b09da617a6a7ea3b7db40cfd4d91eaa9/lib/ex_unit/lib/ex_unit.ex#L194

mix test の多種多様なオプションたち

https://hexdocs.pm/mix/1.12/Mix.Tasks.Test.html#module-command-line-options を見るか、
CLI派の人は mix help test を実行してmix taskのhelpを呼び出すとわかりやすいです。

mix help test

                      mix test                       

Runs the tests for a project.

This task starts the current application, loads up
test/test_helper.exs and then, requires all files
matching the test/**/*_test.exs pattern in parallel.

A list of files and/or directories can be given
after the task name in order to select the files to
run:

    mix test test/some/particular/file_test.exs
    mix test test/some/particular/dir

Tests in umbrella projects can be run from the root
by specifying the full suite path, including
apps/my_app/test, in which case recursive tests for
other child apps will be skipped completely:

    # To run all tests for my_app from the umbrella root
    mix test apps/my_app/test
    
    # To run a given test file on my_app from the umbrella root
    mix test apps/my_app/test/some/particular/file_test.exs

## Understanding test results

When you run your test suite, it prints results as
they run with a summary at the end, as seen below:

    $ mix test
    ...
    
      1) test greets the world (FooTest)
         test/foo_test.exs:5
         Assertion with == failed
         code:  assert Foo.hello() == :world!
         left:  :world
         right: :world!
         stacktrace:
           test/foo_test.exs:6: (test)
    
    ........
    
    Finished in 0.05 seconds (0.00s async, 0.05s sync)
    1 doctest, 11 tests, 1 failure
    
    Randomized with seed 646219

For each test, the test suite will print a dot.
Failed tests are printed immediately in the format
described in the next section.

After all tests run, we print the suite summary. The
first line contains the total time spent on the
suite, followed by how much time was spent on async
tests (defined with use ExUnit.Case, async: true) vs
sync ones:

    Finished in 0.05 seconds (0.00s async, 0.05s sync)

Developers want to minimize the time spent on sync
tests whenever possible, as sync tests run serially
and async tests run concurrently.

Finally, how many tests we have run, how many of
them failed, how many were invalid, etc.

### Understanding test failures

First, it contains the failure counter, followed by
the test name and the module the test was defined:

    1) test greets the world (FooTest)

The next line contains the exact location of the
test in the FILE:LINE format:

    test/foo_test.exs:5

If you want to re-run only this test, all you need
to do is to copy the line above and past it in front
of mix test:

    mix test test/foo_test.exs:5

Then we show the error message, code snippet, and
general information about the failed test:

    Assertion with == failed
    code:  assert Foo.hello() == :world!
    left:  :world
    right: :world!

If your terminal supports coloring (see the 
"Coloring" section below), a diff is typically shown
between left and right sides. Finally, we print the
stacktrace of the failure:

    stacktrace:
      test/foo_test.exs:6: (test)

## Command line options

  • --color - enables color in the output
  • --cover - runs coverage tool. See "Coverage"
    section below
  • --exclude - excludes tests that match the
    filter
  • --export-coverage - the name of the file to
    export coverage results to. Only has an effect
    when used with --cover
  • --failed - runs only tests that failed the
    last time they ran
  • --force - forces compilation regardless of
    modification times
  • --formatter - sets the formatter module that
    will print the results. Defaults to ExUnit's
    built-in CLI formatter
  • --include - includes tests that match the
    filter
  • --listen-on-stdin - runs tests, and then
    listens on stdin. It will re-run tests once a
    newline is received. See the "File system
    watchers" section below
  • --max-cases - sets the maximum number of
    tests running asynchronously. Only tests from
    different modules run in parallel. Defaults to
    twice the number of cores
  • --max-failures - the suite stops evaluating
    tests when this number of test failures is
    reached. It runs all tests if omitted
  • --no-archives-check - does not check
    archives
  • --no-color - disables color in the output
  • --no-compile - does not compile, even if
    files require compilation
  • --no-deps-check - does not check
    dependencies
  • --no-elixir-version-check - does not check
    the Elixir version from mix.exs
  • --no-start - does not start applications
    after compilation
  • --only - runs only tests that match the
    filter
  • --partitions - sets the amount of partitions
    to split tests in. This option requires the
    MIX_TEST_PARTITION environment variable to be
    set. See the "Operating system process
    partitioning" section for more information
  • --preload-modules - preloads all modules
    defined in applications
  • --raise - raises if the test suite failed
  • --seed - seeds the random number generator
    used to randomize the order of tests; --seed 0
    disables randomization so the tests in a single
    file will always be ran in the same order they
    were defined in
  • --slowest - prints timing information for
    the N slowest tests. Automatically sets --trace
    and --preload-modules
  • --stale - runs only tests which reference
    modules that changed since the last time tests
    were ran with --stale. You can read more about
    this option in the "The --stale option" section
    below
  • --timeout - sets the timeout for the tests
  • --trace - runs tests with detailed
    reporting. Automatically sets --max-cases to 1.
    Note that in trace mode test timeouts will be
    ignored as timeout is set to :infinity
  • --warnings-as-errors - (since v1.12.0)
    treats warnings as errors and returns a non-zero
    exit code. This option only applies to test
    files. To treat warnings as errors during
    compilation and during tests, run:
        MIX_ENV=test mix do compile --warnings-as-errors, test --warnings-as-errors


## Configuration

These configurations can be set in the def project
section of your mix.exs:

  • :test_paths - list of paths containing test
    files. Defaults to ["test"] if the test
    directory exists; otherwise, it defaults to [].
    It is expected that all test paths contain a
    test_helper.exs file
  • :test_pattern - a pattern to load test
    files. Defaults to *_test.exs
  • :warn_test_pattern - a pattern to match
    potentially misnamed test files and display a
    warning. Defaults to *_test.ex
  • :test_coverage - a set of options to be
    passed down to the coverage mechanism

## Coloring

Coloring is enabled by default on most Unix
terminals. They are also available on Windows
consoles from Windows 10, although it must be
explicitly enabled for the current user in the
registry by running the following command:

    reg add HKCU\Console /v VirtualTerminalLevel /t REG_DWORD /d 1

After running the command above, you must restart
your current console.

## Filters

ExUnit provides tags and filtering functionality
that allow developers to select which tests to run.
The most common functionality is to exclude some
particular tests from running by default in your
test helper file:

    # Exclude all external tests from running
    ExUnit.configure(exclude: [external: true])

Then, whenever desired, those tests could be
included in the run via the --include option:

    mix test --include external:true

The example above will run all tests that have the
external option set to true. It is also possible to
include all examples that have a given tag,
regardless of its value:

    mix test --include external

Note that all tests are included by default, so
unless they are excluded first (either in the test
helper or via the --exclude option) the --include
option has no effect.

For this reason, Mix also provides an --only option
that excludes all tests and includes only the given
ones:

    mix test --only external

Which is similar to:

    mix test --include external --exclude test

It differs in that the test suite will fail if no
tests are executed when the --only option is used.

In case a single file is being tested, it is
possible to pass one or more specific line numbers
to run only those given tests:

    mix test test/some/particular/file_test.exs:12

Which is equivalent to:

    mix test --exclude test --include line:12 test/some/particular/file_test.exs

Or:

    mix test test/some/particular/file_test.exs:12:24

Which is equivalent to:

    mix test --exclude test --include line:12 --include line:24 test/some/particular/file_test.exs

If a given line starts a describe block, that line
filter runs all tests in it. Otherwise, it runs the
closest test on or before the given line number.

## Coverage

The :test_coverage configuration accepts the
following options:

  • :output - the output directory for cover
    results. Defaults to "cover"
  • :tool - the coverage tool
  • :summary - summary output configuration; can
    be either a boolean or a keyword list. When a
    keyword list is passed, it can specify a
    :threshold, which is a boolean or numeric value
    that enables coloring of code coverage results
    in red or green depending on whether the
    percentage is below or above the specified
    threshold, respectively. Defaults to [threshold:
    90]
  • :export - a file name to export results to
    instead of generating the result on the fly. The
    .coverdata extension is automatically added to
    the given file. This option is automatically set
    via the --export-coverage option or when using
    process partitioning. See mix test.coverage to
    compile a report from multiple exports.
  • :ignore_modules - modules to ignore from
    generating reports and in summaries

By default, a very simple wrapper around OTP's cover
is used as a tool, but it can be overridden as
follows:

    def project() do
      [
        ...
        test_coverage: [tool: CoverModule]
        ...
      ]
    end

CoverModule can be any module that exports start/2,
receiving the compilation path and the test_coverage
options as arguments. It must return either nil or
an anonymous function of zero arity that will be run
after the test suite is done.

## Operating system process partitioning

While ExUnit supports the ability to run tests
concurrently within the same Elixir instance, it is
not always possible to run all tests concurrently.
For example, some tests may rely on global
resources.

For this reason, mix test supports partitioning the
test files across different Elixir instances. This
is done by setting the --partitions option to an
integer, with the number of partitions, and setting
the MIX_TEST_PARTITION environment variable to
control which test partition that particular
instance is running. This can also be useful if you
want to distribute testing across multiple machines.

For example, to split a test suite into 4 partitions
and run them, you would use the following commands:

    MIX_TEST_PARTITION=1 mix test --partitions 4
    MIX_TEST_PARTITION=2 mix test --partitions 4
    MIX_TEST_PARTITION=3 mix test --partitions 4
    MIX_TEST_PARTITION=4 mix test --partitions 4

The test files are sorted upfront in a round-robin
fashion. Note the partition itself is given as an
environment variable so it can be accessed in config
files and test scripts. For example, it can be used
to setup a different database instance per partition
in config/test.exs.

If partitioning is enabled and --cover is used, no
cover reports are generated, as they only contain a
subset of the coverage data. Instead, the coverage
data is exported to files such as
cover/MIX_TEST_PARTITION.coverdata. Once you have
the results of all partitions inside cover/, you can
run mix test.coverage to get the unified report.

## The --stale option

The --stale command line option attempts to run only
the test files which reference modules that have
changed since the last time you ran this task with
--stale.

The first time this task is run with --stale, all
tests are run and a manifest is generated. On
subsequent runs, a test file is marked "stale" if
any modules it references (and any modules those
modules reference, recursively) were modified since
the last run with --stale. A test file is also
marked "stale" if it has been changed since the last
run with --stale.

The --stale option is extremely useful for software
iteration, allowing you to run only the relevant
tests as you perform changes to the codebase.

## File-system watchers

You can integrate mix test with filesystem watchers
through the command line via the --listen-on-stdin
option. For example, you can use fswatch
(https://github.com/emcrisostomo/fswatch) or similar
to emit newlines whenever there is a change, which
will cause your test suite to re-run:

    fswatch lib test | mix test --listen-on-stdin

This can be combined with the --stale option to
re-run only the test files that have changed as well
as the tests that have gone stale due to changes in
lib.

## Aborting the suite

It is possible to abort the test suite with Ctrl+\ ,
which sends a SIGQUIT signal to the Erlang VM.
ExUnit will intercept this signal to show all tests
that have been aborted and print the results
collected so far.

This can be useful in case the suite gets stuck and
you don't want to wait until the timeout times
passes (which defaults to 30 seconds).

Location: /Users/koga/.asdf/installs/elixir/1.12/lib/mix/ebin

オプション部分だけ以下抜粋します。

   • --color - enables color in the output
   • --cover - runs coverage tool. See "Coverage" section below
   • --exclude - excludes tests that match the filter
   • --export-coverage - the name of the file to export coverage results to.
     Only has an effect when used with --cover
   • --failed - runs only tests that failed the last time they ran
   • --force - forces compilation regardless of modification times
   • --formatter - sets the formatter module that will print the results.
     Defaults to ExUnit's built-in CLI formatter
   • --include - includes tests that match the filter
   • --listen-on-stdin - runs tests, and then listens on stdin. It will
     re-run tests once a newline is received. See the "File system watchers"
     section below
   • --max-cases - sets the maximum number of tests running asynchronously.
     Only tests from different modules run in parallel. Defaults to twice the
     number of cores
   • --max-failures - the suite stops evaluating tests when this number of
     test failures is reached. It runs all tests if omitted
   • --no-archives-check - does not check archives
   • --no-color - disables color in the output
   • --no-compile - does not compile, even if files require compilation
   • --no-deps-check - does not check dependencies
   • --no-elixir-version-check - does not check the Elixir version from
     mix.exs
   • --no-start - does not start applications after compilation
   • --only - runs only tests that match the filter
   • --partitions - sets the amount of partitions to split tests in. This
     option requires the MIX_TEST_PARTITION environment variable to be set. See
     the "Operating system process partitioning" section for more information
   • --preload-modules - preloads all modules defined in applications
   • --raise - raises if the test suite failed
   • --seed - seeds the random number generator used to randomize the order
     of tests; --seed 0 disables randomization so the tests in a single file
     will always be ran in the same order they were defined in
   • --slowest - prints timing information for the N slowest tests.
     Automatically sets --trace and --preload-modules
   • --stale - runs only tests which reference modules that changed since
     the last time tests were ran with --stale. You can read more about this
     option in the "The --stale option" section below
   • --timeout - sets the timeout for the tests
   • --trace - runs tests with detailed reporting. Automatically sets
     --max-cases to 1. Note that in trace mode test timeouts will be ignored as
     timeout is set to :infinity
   • --warnings-as-errors - (since v1.12.0) treats warnings as errors and
     returns a non-zero exit code. This option only applies to test files. To
     treat warnings as errors during compilation and during tests, run:
         MIX_ENV=test mix do compile --warnings-as-errors, test --warnings-as-errors

個人的にこれはよく使うというオプションをつらつらと紹介してみます。

直近でエラーになったテストをもう一度流したい -> --failed

mix test --failed

指定した回数エラーになったらテストを落としたい -> --max-failures

1を指定して、1件でもコケたらテスト終了→修正のサイクルを回すときによく使います。

mix test --max-failures 1

(オプションではないが)ファイルの特定行のテストを実行したい

mix test <path>:<line number>

note: test do で囲まれた行であれば何行目を指定してもOK
(長いこと test doの行を明示的に指定しないとダメと思ってました😇

遅いテスト上位N件を調査したい -> --slowest N

mix test --slowest 10

変更の入ったモジュールを対象としたテストだけ実行したい -> --stale

mix test --stale

さらに学ぶ

mix test の動きが把握できたところで、さらにテストを深堀りするためのキーワードをつらつらと書いておきます📚

まとめ

mix test を実行してテストが動作する流れを辿ってみました。よきテストライフを🚀

P.S.

9月、Elixirコミュニティのイベントが続々と開催されています。明日のEDIのイベントでExUnitについて話すつもりで、改めて整理していったら記事になったというオチでした。

両方ともzoomでのリモート参加型イベントです。興味ある方はぜひチェックしてみてください📝

https://fukuokaex.connpass.com/event/224883/

https://fukuokaex.connpass.com/event/224171/

Discussion