🤖
mix testでテストが実行される過程を追ってみる
Elixirで mix new からMix projectを作成すると
test/test_helper.exstest/<project名>_test.exs
といったファイルが自動で作られており、何も意識せずとも mix test を実行すればテストがすでに実行できる状態になっています。
# test/test_helper.exs
ExUnit.start()
# test/sample_test.exs
defmodule SampleTest do
use ExUnit.Case
doctest Sample
test "greets the world" do
assert Sample.hello() == :world
end
end
$ mix test
Compiling 1 file (.ex)
Generated sample app
..
Finished in 0.04 seconds (0.00s async, 0.04s sync)
1 doctest, 1 test, 0 failures
Randomized with seed 331689
これらが何をやっているのか、深堀りしてみたいと思います。
ExUnit
mix testを掘っていく前に、ExUnitについて抑えていきましょう。hexdocsの冒頭に
Unit testing framework for Elixir.
と記載されている通り、ExUnitはElixirのテスティングフレームワークです。
最小構成でテストを実行してみる
テストの環境は mix new や mix phx.new したらついてくるものという印象が強い(？)ですが、Elixir本体に組み込まれているものなので、実はmix projectでなくても実行可能です。以下、hexdocsの例をそのまま持ってきました。
# File: assertion_test.exs
# 1) Start ExUnit.
ExUnit.start()
# 2) Create a new test module (test case) and use "ExUnit.Case".
defmodule AssertionTest do
# 3) Note that we pass "async: true", this runs the test case
# concurrently with other test cases. The individual tests
# within each test case are still run serially.
use ExUnit.Case, async: true
# 4) Use the "test" macro instead of "def" for clarity.
test "the truth" do
assert true
end
end
$ elixir assertion_test.exs
.
Finished in 0.05 seconds (0.05s on load, 0.00s async, 0.00s sync)
1 test, 0 failures
Randomized with seed 950675
-
ExUnit.start/0を実行 -
use ExUnit.caseを記述したモジュールを記述 -
async: trueをオプションとして渡せば並列に実行される -
testマクロを使ってテストを記述
というステップを踏めば、もうテストが実行できます。
ExUnit.Case
use ExUnit.Case
これの記述のおかげで test ...do や describe ... do のようなマクロが使えるようになります。試しにuseの記述を消して実行するとCompileErrorになります。
$ elixir assertion_test.exs
** (CompileError) assertion_test.exs:12: undefined function test/2
内部では ExUnit.Callbacks でsetupやon_exitなどのcallbackが定義されていたり、 ExUnit.Assertions でassertやrefuteなどのmacroが定義されていたりします。これらをひっくるめて use ExUnit.Case さえ呼べばこれらのモジュール群が読み込めるという作りになっています。
# lib/ex_unit/lib/ex_unit/case.ex
@doc false
defmacro __using__(opts) do
unless Process.whereis(ExUnit.Server) do
raise "cannot use ExUnit.Case without starting the ExUnit application, " <>
"please call ExUnit.start() or explicitly start the :ex_unit app"
end
quote do
unless ExUnit.Case.__register__(__MODULE__, unquote(opts)) do
use ExUnit.Callbacks
end
import ExUnit.Callbacks
import ExUnit.Assertions
import ExUnit.Case, only: [describe: 2, test: 1, test: 2, test: 3]
import ExUnit.DocTest
end
end
mixとの統合
elixirでは mix test というMixタスクが標準で提供（コード参考）されており、 mix test を実行すると
-
test/test_helper.exsを読み込み -
test/**/*_test.exsを並列に実行
という順序でテストケースが実行されます。これがこの記事冒頭に書いた2つのファイルであり、最初に読み込まれる test_helper.exs に ExUnit.start() が記載されているため、テストが実行できるというわけです。
「じゃExUnit.startは何をやっているの？」まではちょっと大変ですね。この記事のスコープからはいったん外してヨシということで。興味がある人はぜひ辿ってみて詳細教えてください。
mix test の多種多様なオプションたち
https://hexdocs.pm/mix/1.12/Mix.Tasks.Test.html#module-command-line-options を見るか、
CLI派の人は mix help test を実行してmix taskのhelpを呼び出すとわかりやすいです。
mix help test
mix test
Runs the tests for a project.
This task starts the current application, loads up
test/test_helper.exs and then, requires all files
matching the test/**/*_test.exs pattern in parallel.
A list of files and/or directories can be given
after the task name in order to select the files to
run:
mix test test/some/particular/file_test.exs
mix test test/some/particular/dir
Tests in umbrella projects can be run from the root
by specifying the full suite path, including
apps/my_app/test, in which case recursive tests for
other child apps will be skipped completely:
# To run all tests for my_app from the umbrella root
mix test apps/my_app/test
# To run a given test file on my_app from the umbrella root
mix test apps/my_app/test/some/particular/file_test.exs
## Understanding test results
When you run your test suite, it prints results as
they run with a summary at the end, as seen below:
$ mix test
...
1) test greets the world (FooTest)
test/foo_test.exs:5
Assertion with == failed
code: assert Foo.hello() == :world!
left: :world
right: :world!
stacktrace:
test/foo_test.exs:6: (test)
........
Finished in 0.05 seconds (0.00s async, 0.05s sync)
1 doctest, 11 tests, 1 failure
Randomized with seed 646219
For each test, the test suite will print a dot.
Failed tests are printed immediately in the format
described in the next section.
After all tests run, we print the suite summary. The
first line contains the total time spent on the
suite, followed by how much time was spent on async
tests (defined with use ExUnit.Case, async: true) vs
sync ones:
Finished in 0.05 seconds (0.00s async, 0.05s sync)
Developers want to minimize the time spent on sync
tests whenever possible, as sync tests run serially
and async tests run concurrently.
Finally, how many tests we have run, how many of
them failed, how many were invalid, etc.
### Understanding test failures
First, it contains the failure counter, followed by
the test name and the module the test was defined:
1) test greets the world (FooTest)
The next line contains the exact location of the
test in the FILE:LINE format:
test/foo_test.exs:5
If you want to re-run only this test, all you need
to do is to copy the line above and past it in front
of mix test:
mix test test/foo_test.exs:5
Then we show the error message, code snippet, and
general information about the failed test:
Assertion with == failed
code: assert Foo.hello() == :world!
left: :world
right: :world!
If your terminal supports coloring (see the
"Coloring" section below), a diff is typically shown
between left and right sides. Finally, we print the
stacktrace of the failure:
stacktrace:
test/foo_test.exs:6: (test)
## Command line options
• --color - enables color in the output
• --cover - runs coverage tool. See "Coverage"
section below
• --exclude - excludes tests that match the
filter
• --export-coverage - the name of the file to
export coverage results to. Only has an effect
when used with --cover
• --failed - runs only tests that failed the
last time they ran
• --force - forces compilation regardless of
modification times
• --formatter - sets the formatter module that
will print the results. Defaults to ExUnit's
built-in CLI formatter
• --include - includes tests that match the
filter
• --listen-on-stdin - runs tests, and then
listens on stdin. It will re-run tests once a
newline is received. See the "File system
watchers" section below
• --max-cases - sets the maximum number of
tests running asynchronously. Only tests from
different modules run in parallel. Defaults to
twice the number of cores
• --max-failures - the suite stops evaluating
tests when this number of test failures is
reached. It runs all tests if omitted
• --no-archives-check - does not check
archives
• --no-color - disables color in the output
• --no-compile - does not compile, even if
files require compilation
• --no-deps-check - does not check
dependencies
• --no-elixir-version-check - does not check
the Elixir version from mix.exs
• --no-start - does not start applications
after compilation
• --only - runs only tests that match the
filter
• --partitions - sets the amount of partitions
to split tests in. This option requires the
MIX_TEST_PARTITION environment variable to be
set. See the "Operating system process
partitioning" section for more information
• --preload-modules - preloads all modules
defined in applications
• --raise - raises if the test suite failed
• --seed - seeds the random number generator
used to randomize the order of tests; --seed 0
disables randomization so the tests in a single
file will always be ran in the same order they
were defined in
• --slowest - prints timing information for
the N slowest tests. Automatically sets --trace
and --preload-modules
• --stale - runs only tests which reference
modules that changed since the last time tests
were ran with --stale. You can read more about
this option in the "The --stale option" section
below
• --timeout - sets the timeout for the tests
• --trace - runs tests with detailed
reporting. Automatically sets --max-cases to 1.
Note that in trace mode test timeouts will be
ignored as timeout is set to :infinity
• --warnings-as-errors - (since v1.12.0)
treats warnings as errors and returns a non-zero
exit code. This option only applies to test
files. To treat warnings as errors during
compilation and during tests, run:
MIX_ENV=test mix do compile --warnings-as-errors, test --warnings-as-errors
## Configuration
These configurations can be set in the def project
section of your mix.exs:
• :test_paths - list of paths containing test
files. Defaults to ["test"] if the test
directory exists; otherwise, it defaults to [].
It is expected that all test paths contain a
test_helper.exs file
• :test_pattern - a pattern to load test
files. Defaults to *_test.exs
• :warn_test_pattern - a pattern to match
potentially misnamed test files and display a
warning. Defaults to *_test.ex
• :test_coverage - a set of options to be
passed down to the coverage mechanism
## Coloring
Coloring is enabled by default on most Unix
terminals. They are also available on Windows
consoles from Windows 10, although it must be
explicitly enabled for the current user in the
registry by running the following command:
reg add HKCU\Console /v VirtualTerminalLevel /t REG_DWORD /d 1
After running the command above, you must restart
your current console.
## Filters
ExUnit provides tags and filtering functionality
that allow developers to select which tests to run.
The most common functionality is to exclude some
particular tests from running by default in your
test helper file:
# Exclude all external tests from running
ExUnit.configure(exclude: [external: true])
Then, whenever desired, those tests could be
included in the run via the --include option:
mix test --include external:true
The example above will run all tests that have the
external option set to true. It is also possible to
include all examples that have a given tag,
regardless of its value:
mix test --include external
Note that all tests are included by default, so
unless they are excluded first (either in the test
helper or via the --exclude option) the --include
option has no effect.
For this reason, Mix also provides an --only option
that excludes all tests and includes only the given
ones:
mix test --only external
Which is similar to:
mix test --include external --exclude test
It differs in that the test suite will fail if no
tests are executed when the --only option is used.
In case a single file is being tested, it is
possible to pass one or more specific line numbers
to run only those given tests:
mix test test/some/particular/file_test.exs:12
Which is equivalent to:
mix test --exclude test --include line:12 test/some/particular/file_test.exs
Or:
mix test test/some/particular/file_test.exs:12:24
Which is equivalent to:
mix test --exclude test --include line:12 --include line:24 test/some/particular/file_test.exs
If a given line starts a describe block, that line
filter runs all tests in it. Otherwise, it runs the
closest test on or before the given line number.
## Coverage
The :test_coverage configuration accepts the
following options:
• :output - the output directory for cover
results. Defaults to "cover"
• :tool - the coverage tool
• :summary - summary output configuration; can
be either a boolean or a keyword list. When a
keyword list is passed, it can specify a
:threshold, which is a boolean or numeric value
that enables coloring of code coverage results
in red or green depending on whether the
percentage is below or above the specified
threshold, respectively. Defaults to [threshold:
90]
• :export - a file name to export results to
instead of generating the result on the fly. The
.coverdata extension is automatically added to
the given file. This option is automatically set
via the --export-coverage option or when using
process partitioning. See mix test.coverage to
compile a report from multiple exports.
• :ignore_modules - modules to ignore from
generating reports and in summaries
By default, a very simple wrapper around OTP's cover
is used as a tool, but it can be overridden as
follows:
def project() do
[
...
test_coverage: [tool: CoverModule]
...
]
end
CoverModule can be any module that exports start/2,
receiving the compilation path and the test_coverage
options as arguments. It must return either nil or
an anonymous function of zero arity that will be run
after the test suite is done.
## Operating system process partitioning
While ExUnit supports the ability to run tests
concurrently within the same Elixir instance, it is
not always possible to run all tests concurrently.
For example, some tests may rely on global
resources.
For this reason, mix test supports partitioning the
test files across different Elixir instances. This
is done by setting the --partitions option to an
integer, with the number of partitions, and setting
the MIX_TEST_PARTITION environment variable to
control which test partition that particular
instance is running. This can also be useful if you
want to distribute testing across multiple machines.
For example, to split a test suite into 4 partitions
and run them, you would use the following commands:
MIX_TEST_PARTITION=1 mix test --partitions 4
MIX_TEST_PARTITION=2 mix test --partitions 4
MIX_TEST_PARTITION=3 mix test --partitions 4
MIX_TEST_PARTITION=4 mix test --partitions 4
The test files are sorted upfront in a round-robin
fashion. Note the partition itself is given as an
environment variable so it can be accessed in config
files and test scripts. For example, it can be used
to setup a different database instance per partition
in config/test.exs.
If partitioning is enabled and --cover is used, no
cover reports are generated, as they only contain a
subset of the coverage data. Instead, the coverage
data is exported to files such as
cover/MIX_TEST_PARTITION.coverdata. Once you have
the results of all partitions inside cover/, you can
run mix test.coverage to get the unified report.
## The --stale option
The --stale command line option attempts to run only
the test files which reference modules that have
changed since the last time you ran this task with
--stale.
The first time this task is run with --stale, all
tests are run and a manifest is generated. On
subsequent runs, a test file is marked "stale" if
any modules it references (and any modules those
modules reference, recursively) were modified since
the last run with --stale. A test file is also
marked "stale" if it has been changed since the last
run with --stale.
The --stale option is extremely useful for software
iteration, allowing you to run only the relevant
tests as you perform changes to the codebase.
## File-system watchers
You can integrate mix test with filesystem watchers
through the command line via the --listen-on-stdin
option. For example, you can use fswatch
(https://github.com/emcrisostomo/fswatch) or similar
to emit newlines whenever there is a change, which
will cause your test suite to re-run:
fswatch lib test | mix test --listen-on-stdin
This can be combined with the --stale option to
re-run only the test files that have changed as well
as the tests that have gone stale due to changes in
lib.
## Aborting the suite
It is possible to abort the test suite with Ctrl+\ ,
which sends a SIGQUIT signal to the Erlang VM.
ExUnit will intercept this signal to show all tests
that have been aborted and print the results
collected so far.
This can be useful in case the suite gets stuck and
you don't want to wait until the timeout times
passes (which defaults to 30 seconds).
Location: /Users/koga/.asdf/installs/elixir/1.12/lib/mix/ebin
オプション部分だけ以下抜粋します。
• --color - enables color in the output
• --cover - runs coverage tool. See "Coverage" section below
• --exclude - excludes tests that match the filter
• --export-coverage - the name of the file to export coverage results to.
Only has an effect when used with --cover
• --failed - runs only tests that failed the last time they ran
• --force - forces compilation regardless of modification times
• --formatter - sets the formatter module that will print the results.
Defaults to ExUnit's built-in CLI formatter
• --include - includes tests that match the filter
• --listen-on-stdin - runs tests, and then listens on stdin. It will
re-run tests once a newline is received. See the "File system watchers"
section below
• --max-cases - sets the maximum number of tests running asynchronously.
Only tests from different modules run in parallel. Defaults to twice the
number of cores
• --max-failures - the suite stops evaluating tests when this number of
test failures is reached. It runs all tests if omitted
• --no-archives-check - does not check archives
• --no-color - disables color in the output
• --no-compile - does not compile, even if files require compilation
• --no-deps-check - does not check dependencies
• --no-elixir-version-check - does not check the Elixir version from
mix.exs
• --no-start - does not start applications after compilation
• --only - runs only tests that match the filter
• --partitions - sets the amount of partitions to split tests in. This
option requires the MIX_TEST_PARTITION environment variable to be set. See
the "Operating system process partitioning" section for more information
• --preload-modules - preloads all modules defined in applications
• --raise - raises if the test suite failed
• --seed - seeds the random number generator used to randomize the order
of tests; --seed 0 disables randomization so the tests in a single file
will always be ran in the same order they were defined in
• --slowest - prints timing information for the N slowest tests.
Automatically sets --trace and --preload-modules
• --stale - runs only tests which reference modules that changed since
the last time tests were ran with --stale. You can read more about this
option in the "The --stale option" section below
• --timeout - sets the timeout for the tests
• --trace - runs tests with detailed reporting. Automatically sets
--max-cases to 1. Note that in trace mode test timeouts will be ignored as
timeout is set to :infinity
• --warnings-as-errors - (since v1.12.0) treats warnings as errors and
returns a non-zero exit code. This option only applies to test files. To
treat warnings as errors during compilation and during tests, run:
MIX_ENV=test mix do compile --warnings-as-errors, test --warnings-as-errors
個人的にこれはよく使うというオプションをつらつらと紹介してみます。
直近でエラーになったテストをもう一度流したい -> --failed
mix test --failed
指定した回数エラーになったらテストを落としたい -> --max-failures
1を指定して、1件でもコケたらテスト終了→修正のサイクルを回すときによく使います。
mix test --max-failures 1
(オプションではないが)ファイルの特定行のテストを実行したい
mix test <path>:<line number>
note: test do で囲まれた行であれば何行目を指定してもOK
（長いこと test doの行を明示的に指定しないとダメと思ってました😇
遅いテスト上位N件を調査したい -> --slowest N
mix test --slowest 10
変更の入ったモジュールを対象としたテストだけ実行したい -> --stale
mix test --stale
さらに学ぶ
mix test の動きが把握できたところで、さらにテストを深堀りするためのキーワードをつらつらと書いておきます📚
- setup, on_exitなどのcallbackを活用してtestを記述する
-
@tag: :skipなどタグを活用して柔軟にテストを実行する- 重たいテストは毎回のciでは無視し、特定ブランチのciだけでは実行させる等
- 副作用がないテストはdoctestでドキュメントとともに記述する
- CaseTemplateを利用して独自のCaseを定義する
- https://hexdocs.pm/ex_unit/1.12/ExUnit.CaseTemplate.html
- PhoenixのConnCaseやDataCaseが好例
- ExMachinaやFakerを利用してテストデータの作成をスムーズに行う
- moxなど、mockライブラリの導入
まとめ
mix test を実行してテストが動作する流れを辿ってみました。よきテストライフを🚀
P.S.
9月、Elixirコミュニティのイベントが続々と開催されています。明日のEDIのイベントでExUnitについて話すつもりで、改めて整理していったら記事になったというオチでした。
両方ともzoomでのリモート参加型イベントです。興味ある方はぜひチェックしてみてください📝
Discussion