iTranslated by AI

The content below is an AI-generated translation. This is an experimental feature, and may contain errors. View original article
🎴

Building a Property-Based Testing Library

に公開
Test
Dart
PBT
プロパティベーステスト
tech

I created a property-based testing library for Dart called kiri-check. I had been trying out a library for Dart called Glados for a while, but I felt it was a bit lacking in features and didn't seem to be very actively maintained, so I decided to try building my own for learning purposes.

In this article, I would like to write about how to use the library simply and the insights I gained through its implementation. I won't explain property-based testing in detail here, so please refer to resources like Practical Property-Based Testing.

By the way, the "kiri" in kiri-check comes from the Japanese phrase "kiri ga nai" (endless/limitless).

Features of kiri-check

I have tried to avoid creating a unique user experience as much as possible. There are no "kiri-check exclusive features!"

  • It uses the test package, so you can use it with the same feeling as your usual tests. Also, you can add property-based tests without changing existing test code.
  • It has a simple API similar to major property-based testing libraries. I mainly referred to QuickCheck, PropEr, Hypothesis, and jqwik.
  • Customizing the exploration method is easy. You can specify data ranges, increase or decrease the number of trials, and prioritize edge cases.
  • Supports generation of Unicode strings. It also supports Unicode categories, allowing the generation of strings composed of specific character types or code points.
  • Supports time zones and imaginary dates. Imaginary dates refer to non-existent dates like February 29th in a non-leap year or April 31st.

I also plan to implement the following features. I want them myself, so I'd like to implement them soon.

  • Stateful testing
  • Targeted properties

Installation

You can install it via pub.dev. Run the following command or add it manually to your pubspec.yaml.

dart pub add kiri_check

Quick Start

As a simple example, we will implement a test for FizzBuzz. Of course, I have provided a FizzBuzz implementation, but please feel free to implement and test it yourself.

First, prepare a new Dart project and install kiri-check. Next, create a FizzBuzz test file under the test directory. The test code is as follows.

test/fizzbuzz_test.dart:

Object fizzbuzzGood(int n) {
  if (n % 3 == 0 && n % 5 == 0) {
    return 'FizzBuzz';
  } else if (n % 3 == 0) {
    return 'Fizz';
  } else if (n % 5 == 0) {
    return 'Buzz';
  } else {
    return n;
  }
}

A test for this function can be implemented as follows:

void main() {
  property('FizzBuzz (good)', () {
    forAll(
      integer(min: 0, max: 100),
      (n) {
        final result = fizzbuzzGood(n);
        if (n % 15 == 0) {
          expect(result, 'FizzBuzz');
        } else if (n % 3 == 0) {
          expect(result, 'Fizz');
        } else if (n % 5 == 0) {
          expect(result, 'Buzz');
        } else {
          expect(result, n);
        }
      },
    );
  });
}

The block passed to property is the test content. property is a wrapper for test and accepts the same arguments as test. Excluding forAll, it can be used in the same way as test. You can also use methods like expect for checking values, just like in test.

You implement specific tests within the block passed to forAll. forAll takes a generator (i.e., arbitrary) as an argument and executes the block by passing values (almost) randomly generated by the generator.

integer is a function that creates a generator for integers. You specify the minimum and maximum values with the min and max arguments. When the test is executed, integers are generated by integer, and the block is executed repeatedly for that many counts. The default number of generations is 100.

Since FizzBuzz has simple specifications, the test content will be almost the same as the implementation. However, I've slightly changed the conditional expressions for multiples of 3 and 5 between the FizzBuzz implementation and the test code. While the FizzBuzz implementation combines two divisions by 3 and 5, the test code uses a constant to reduce the chance of mistakes. Even if it's simple, using the exact same code might cause you to miss bugs because the test would pass anyway. Be careful, as this would defeat the purpose of property testing.

Running the Tests

You can run it in the same way as test. Run it using the dart test subcommand or via your IDE.

dart test fizzbuzz_test.dart

Output:

00:00 +1: All tests passed!

All tests passed successfully. However, just seeing this doesn't tell us if the test was truly meaningful, so let's display the generated values. To show test details including the generated values, specify Verbosity.verbose for KiriCheck.verbosity.

void main() {
  KiriCheck.verbosity = Verbosity.verbose;
  ...

Execution results:

00:00 +0: FizzBuzz (good)                                                                                          
Max tries: 100
Max examples: 100
Random seed: null
Generation policy: GenerationPolicy.auto
Edge case policy: EdgeCasePolicy.mixin
Shrinking policy: null
Trying example: 0
Trying example: 5
Trying example: 73
Trying example: 33
Trying example: 4
(omitted)
Trying example: 72
Trying example: 7
Trying example: 35
Trying example: 31
Trying example: 1
No falsifying examples found (100 tests passed)

00:00 +1: All tests passed!
Complete output
00:00 +0: FizzBuzz (good)                                                                                          
Max tries: 100
Max examples: 100
Random seed: null
Generation policy: GenerationPolicy.auto
Edge case policy: EdgeCasePolicy.mixin
Shrinking policy: null
Trying example: 0
Trying example: 5
Trying example: 73
Trying example: 33
Trying example: 4
Trying example: 0
Trying example: 62
Trying example: 100
Trying example: 85
Trying example: 19
Trying example: 5
Trying example: 3
Trying example: 0
Trying example: 32
Trying example: 2
Trying example: 8
Trying example: 38
Trying example: 71
Trying example: 71
Trying example: 30
Trying example: 25
Trying example: 9
Trying example: 2
Trying example: 62
Trying example: 32
Trying example: 99
Trying example: 36
Trying example: 4
Trying example: 21
Trying example: 54
Trying example: 27
Trying example: 36
Trying example: 97
Trying example: 22
Trying example: 8
Trying example: 78
Trying example: 4
Trying example: 83
Trying example: 100
Trying example: 96
Trying example: 5
Trying example: 0
Trying example: 1
Trying example: 9
Trying example: 89
Trying example: 96
Trying example: 45
Trying example: 67
Trying example: 29
Trying example: 85
Trying example: 55
Trying example: 72
Trying example: 1
Trying example: 65
Trying example: 23
Trying example: 0
Trying example: 76
Trying example: 18
Trying example: 30
Trying example: 34
Trying example: 4
Trying example: 0
Trying example: 24
Trying example: 29
Trying example: 13
Trying example: 83
Trying example: 65
Trying example: 6
Trying example: 52
Trying example: 18
Trying example: 99
Trying example: 79
Trying example: 36
Trying example: 2
Trying example: 92
Trying example: 76
Trying example: 9
Trying example: 56
Trying example: 5
Trying example: 95
Trying example: 5
Trying example: 29
Trying example: 87
Trying example: 55
Trying example: 6
Trying example: 0
Trying example: 81
Trying example: 52
Trying example: 75
Trying example: 100
Trying example: 100
Trying example: 1
Trying example: 14
Trying example: 77
Trying example: 0
Trying example: 72
Trying example: 7
Trying example: 35
Trying example: 31
Trying example: 1
No falsifying examples found (100 tests passed)

00:00 +1: All tests passed!

Trying example: ... indicates the value actually used for the test. No falsifying examples found indicates that no errors occurred with any of the values. Some values are duplicated, but this is intentional and not an issue. Since the outcome of a test with the same value might differ between the first and subsequent runs, duplicates are allowed by design.

Trying to Make a Test Fail

Next, we'll intentionally make a mistake in the implementation to see the test fail. Let's define a new function with slightly altered conditional branches.

Object fizzbuzzBad(int n) {
  if (n % 3 == 0) {
    return 'Fizz';
  } else if (n % 5 == 0) {
    return 'Buzz';
  } else if (n % 3 == 0 && n % 5 == 0) {
    return 'FizzBuzz';
  } else {
    return n;
  }
}

In this code, "FizzBuzz" will not be output for multiples of 15, including 0. For the sake of explanation, I want to exclude 0, so please add the following test with the minimum generated value set to 1.

property('bad', () {
  forAll(
    integer(min: 1, max: 100),
    (n) {
      final result = fizzbuzzBad(n);
      if (n % 15 == 0) {
        expect(result, 'FizzBuzz');
      } else if (n % 3 == 0) {
        expect(result, 'Fizz');
      } else if (n % 5 == 0) {
        expect(result, 'Buzz');
      } else {
        expect(result, n);
      }
    },
    verbose: true,
  );
});

Execution results:

00:00 +1: FizzBuzz (bad)                                                                                           
(omitted)
Trying example: 95
Falsifying example: 45
Shrinking step 1
Shrunk example to: 41
Shrunk example to: 37
Shrunk example to: 33
(omitted)
Shrunk example to: 3
Shrunk example to: 2
Shrunk example to: 1
Shrinking step 4
Minimal failing example: 15
00:00 +1 -1: FizzBuzz (bad) [E]

  Falsifying example: 15
Complete output
00:00 +1: FizzBuzz (bad)                                                                                           
Max tries: 100
Max examples: 100
Random seed: null
Generation policy: GenerationPolicy.auto
Edge case policy: EdgeCasePolicy.mixin
Shrinking policy: null
Trying example: 1
Trying example: 100
Trying example: 71
Trying example: 10
Trying example: 72
Trying example: 81
Trying example: 46
Trying example: 68
Trying example: 2
Trying example: 74
Trying example: 95
Falsifying example: 45
Shrinking step 1
Shrunk example to: 41
Shrunk example to: 37
Shrunk example to: 33
Shrunk example to: 29
Shrunk example to: 25
Shrunk example to: 21
Shrunk example to: 17
Shrunk example to: 13
Shrunk example to: 9
Shrunk example to: 1
Shrinking step 2
Shrunk example to: 43
Shrunk example to: 41
Shrunk example to: 39
Shrunk example to: 37
Shrunk example to: 35
Shrunk example to: 33
Shrunk example to: 31
Shrunk example to: 29
Shrunk example to: 27
Shrunk example to: 25
Shrunk example to: 23
Shrunk example to: 21
Shrunk example to: 19
Shrunk example to: 17
Shrunk example to: 15
Shrunk example to: 13
Shrunk example to: 11
Shrunk example to: 9
Shrunk example to: 7
Shrunk example to: 1
Shrinking step 3
Shrunk example to: 14
Shrunk example to: 13
Shrunk example to: 12
Shrunk example to: 11
Shrunk example to: 10
Shrunk example to: 9
Shrunk example to: 8
Shrunk example to: 7
Shrunk example to: 6
Shrunk example to: 5
Shrunk example to: 4
Shrunk example to: 3
Shrunk example to: 2
Shrunk example to: 1
Shrinking step 4
Minimal failing example: 15
00:00 +1 -1: FizzBuzz (bad) [E]
  Falsifying example: 15
  package:kiri_check/src/property.dart 103:7   Property.check
  package:kiri_check/src/property.dart 398:16  PropertyTestRunner.run
  package:kiri_check/src/property.dart 376:25  PropertyTestManager.runTest
  package:kiri_check/src/property.dart 337:27  PropertyTest.register.body

Falsifying example: 45 indicates that an error occurred when the value was 45. In property-based testing, when an error is found, the process repeatedly executes tests while reducing the value toward a base value (0 in the case of integer()) to find the minimum value that triggers the error (the value closest to the base value). This process is called shrinking.

Shrunk example to: ... shows the values tested during the shrinking process. Shrink values are not random; they are generated in as reproducible a way as possible. If an error occurs with a shrink value, new shrink values are generated based on that value and tested. Essentially, the interval between shrink values narrows each time they are regenerated.

The final Minimal failing example: 15 is the minimum error value found during shrinking. Although the first error occurred at 45, shrinking revealed that an error occurs at 15. This saved the effort of debugging a difference of 30.

In this case, the expected minimum value was found, but the true minimum value is not always guaranteed to be found. To find the minimum error value efficiently, techniques like specifying the range of generated data are necessary.

Complete Code

Object fizzbuzzGood(int n) {
  if (n % 3 == 0 && n % 5 == 0) {
    return 'FizzBuzz';
  } else if (n % 3 == 0) {
    return 'Fizz';
  } else if (n % 5 == 0) {
    return 'Buzz';
  } else {
    return n;
  }
}

Object fizzbuzzBad(int n) {
  if (n % 3 == 0) {
    return 'Fizz';
  } else if (n % 5 == 0) {
    return 'Buzz';
  } else if (n % 3 == 0 && n % 5 == 0) {
    return 'FizzBuzz';
  } else {
    return n;
  }
}

property('good', () {
  forAll(
    integer(),
    (n) {
      final result = fizzbuzzGood(n);
      if (n % 15 == 0) {
        expect(result, 'FizzBuzz');
      } else if (n % 3 == 0) {
        expect(result, 'Fizz');
      } else if (n % 5 == 0) {
        expect(result, 'Buzz');
      } else {
        expect(result, n);
      }
    },
  );
});

property('bad', () {
  forAll(
    integer(max: 10000),
    (n) {
      final result = fizzbuzzBad(n);
      if (n % 15 == 0) {
        expect(result, 'FizzBuzz');
      } else if (n % 3 == 0) {
        expect(result, 'Fizz');
      } else if (n % 5 == 0) {
        expect(result, 'Buzz');
      } else {
        expect(result, n);
      }
    },
  );
});

property('bad with range greater than 0', () {
  forAll(
    integer(min: 100),
    (n) {
      final result = fizzbuzzBad(n);
      if (n % 15 == 0) {
        expect(result, 'FizzBuzz');
      } else if (n % 3 == 0) {
        expect(result, 'Fizz');
      } else if (n % 5 == 0) {
        expect(result, 'Buzz');
      } else {
        expect(result, n);
      }
    },
    ignoreFalsify: true,
    onFalsify: (example) {
      expect(example, 105);
    },
    verbose: true,
  );
});

Other Generators

kiri-check provides the following generators.

Basic Data Types

  • boolean: Generates true or false
  • null_: Generates only null
  • integer: Generates an int
  • string: Generates a string. You can specify character codes (ASCII, UTF-8, UTF-16)
  • runes: Generates Runes. The content is the same as string
  • datetime: Generates DateTime (date and time). You can specify the time zone
  • date: Generates DateTime (date only). You can specify the time zone
  • time: Generates DateTime (time only)
  • constant: Generates only the specified value
  • constantFrom: Generates one of the values from the specified list
  • list: Generates a list of values generated by the specified generator
  • map: Generates a map with keys and values from their respective generators
  • set: Generates a set of values generated by the specified generator

Combinations

  • oneOf: Generates a value using one of several generators
  • frequency: Generates a value by specifying the frequency of use for multiple generators
  • combine: Provides a combination of values generated by all specified generators. Up to 8 generators can be specified
  • recursive: Generates values with recursive structures (such as tree structures)
  • deck: Allows the use of arbitrary generators within the test block

Value Transformation

You can transform the generated values by using the following methods on the generators. If an error occurs with the transformed value, shrinking is performed on the value before the transformation.

  • map: Converts the generated value into another value
  • flatMap: Generates a new generator using the generated value
  • filter: Executes the test only on generated values that satisfy the specified condition

Dynamically Selecting Arbitrary Generators

There are cases where you want to choose the generator to use based on the state during execution. To dynamically select an arbitrary generator, use the special generator deck.

As an example, a test that generates a random integer and then generates a floating-point number if that value is even, or a string if it is odd, can be implemented as follows:

property('deck example', () {
  forAll(deck(),
    (d) {
      // Generate a value with an integer generator
      final i = d.draw(integer());
      // Dynamically select the generator to use
      final value = i.isEven() ? d.draw(float()) : d.draw(string());
      ...
    },
  );

The deck generator produces an object with a draw method that executes an arbitrary generator. By passing a generator to draw, it produces a random value using that generator. This allows for more flexible test implementations than using the map method or the combine generator.

Questioning "Random"

"In property-based testing, functional tests can be performed with input values that the generator randomly produces." I have also only recently started working with property-based testing, and even after reading books and using libraries, my understanding was roughly at this level.

Let's check the data actually generated using the Python property-based testing library Hypothesis. Install Hypothesis and run the following code:

from hypothesis import given
from hypothesis import strategies as st

@given(st.integers())
def print_int(x):
    print(x)

print_int()

In Hypothesis, a "strategy" is the component corresponding to a generator. By specifying a strategy with the @given annotation, the generated values are passed to the function.

When you run this code, 100 integers are displayed (some omitted):

0
-2088329321
-107
...
19138
-16932
0
...
22587735
-4
-4
...
15820023
-17039
1132181008727235737
Complete output
0
-2088329321
-107
30540
31590
-126646556799261029466118727496400491262
-59
120
-18364
-2553
12947
-395375784
-395375784
-1698126065338195494
-9933044
-8427408
-21
22245
-86
19138
-16932
0
-88
22587735
-4
-4
10257
17
0
43
-26883
105
24727
20421
-79
26738
-1752309874
1750826043390091266
-407645954
7
1890
7
16779
-7834
-7834
-30
0
-66
-5196
30743
120
-3513428102538797467
-818033726
1110720208
25215540
4751
-90
-90
3
-816
3
85
21876
85
-14
-14
-118952745846941223428316047197423712450
217
25941
21866
21866
21866
27207
18201
25
16532
1453724087561439426
338471515
-1453724089254237319
9710762
-22494
-87
38
6872
-26
0
-100
-25839
-100
20672
0
1
-77
1
1
-292
-1902442358
15820023
-17039
1132181008727235737

Try running it several times and checking the generated values.

2nd time:

0
1758
-29693
12000
5445
...

3rd time:

0
1440722228
83
29461
-11

Looking through them, we can see the following:

  • The first value is 0.
  • The same values are repeated (and frequently).
  • Many values are 2 digits or less, followed by many values around 20,000.
  • (Probably) 64-bit or less.
    • There is no limit to the length of the Python int type, so the maximum value depends on the machine (details can be obtained via sys.int_info).
  • (Probably) Minimum and maximum values are not included.

Particularly the first two, "The first value is 0" and "The same values are repeated," might be unexpected. Also, there are limits on the minimum and maximum values (default behavior, can be specified), and for 64-bit, the minimum and maximum values -9223372036854775808 and 9223372036854775807 are not included.

Less Biased Random Data

Let's compare this with values generated by a standard random number API. Let's try generating ten 64-bit values.

>>> for i in range(0, 10): print(random.randint(-(1<<63), 1<<63))
... 
8310784346635777773
-3973264716979059331
-7043924164725946517
-4502560362736688420
1723727242751946619
-7512053481029123229
4262257813441119981
2084711476245707027
3170151199645919677
-9087175782144334735

We ended up with values that bear no resemblance to the values generated by Hypothesis. This is expected, as the range of 64-bit integers is vast. The probability of generating a positive or negative value with two digits or less is "198 out of 18,446,744,073,709,551,616". The likelihood of generating such a small value within just 100 attempts is far lower than the probability of winning the first prize in the Nenmatsu Jumbo (year-end) lottery (1 in 20,000,000). In other words, the values generated by Hypothesis are not simply random.

Should Generated Values Be Random?

In fact, randomness is discussed in the Hypothesis documentation.

Quote:

One of the things that is often concerning for people using randomized testing is the question of how to reproduce failing test cases.

It is better to think about the data Hypothesis generates as being arbitrary, rather than random. We deliberately generate any valid data that seems likely to cause errors, so you shouldn’t rely on any expected distribution of or relationships between generated data. You can read about “swarm testing” and “coverage guided fuzzing” if you’re interested, because you don’t need to know for Hypothesis!

Translation:

One of the common concerns for those using randomized testing is the question of how to reproduce failing test cases.

It is better to think about the data Hypothesis generates as being arbitrary, rather than random. We deliberately generate any valid data that seems likely to cause errors, so you shouldn't rely on any expected distribution of or relationships between generated data. You can read about "swarm testing" and "coverage-guided fuzzing" if you're interested, but you don't need to know for Hypothesis!

As it turns out, the values generated by Hypothesis are not completely random. What's crucial for testing diverse data is bug detection and test reproducibility. If the 64-bit integer data were completely random, a 100-value test would not be guaranteed to trigger an error. Even if you were lucky enough to find one, it might not be found in the next run. Furthermore, if the value is large, there's a risk that shrinking might fail to find the minimal failing example.

The same applies to data duplication. Sometimes the success or failure of a test can differ even when using the same data. If such a bug exists, discarding duplicate data prematurely might prevent you from discovering it, even if you expand the data range.

Effective testing requires careful thought about the data to be tested and customization of the data content. If the data range is restricted but still too large, it may be necessary to ensure reproducibility by, for example, saving the random seed when an error is found.

Precision of Random Numbers

Regarding the precision of random numbers, my impression after building the library is that "higher is always better, but even slightly lower precision isn't a critical issue." Lower precision increases duplicates, but this can be mitigated by narrowing the data range and increasing the number of trials.

Modern languages often use the Mersenne Twister in their standard libraries, which is generally sufficient.

Data Generation Algorithms

There doesn't seem to be a specific standard for data generation algorithms for generators or shrinking. Even looking at the documentation of other libraries, I haven't seen any detailed descriptions of the generation algorithms used. While there are cases where exploration algorithms like simulated annealing are used in PropEr's targeted properties, special methods didn't seem particularly necessary. Basically, it's probably best not to rely too much on default generation methods and instead carefully consider and explicitly specify the data that should be tested.

Shrinking

Basic Flow

Shrinking is the process of finding the minimal failing value. When an error occurs during testing, shrinking begins, and testing continues by generating new values based on the value that caused the error. When errors no longer occur, the last value that caused an error is treated as the minimum value, and the test is terminated.

For example, if an integer generator fails with the value 10, it will repeatedly try values of 9 or less, and when errors no longer occur, the last value that caused an error is taken as the minimum value.

Users can then perform debugging by referring to this minimum value. The smaller the minimum value, the easier it becomes to identify the cause of the error. If the minimum value is 10 versus 10,000,000, debugging 10 will likely be easier because it saves the cost of investigating the range of 9,999,990. (Of course, other causes of errors might be hidden in this range. Shrinking is just a reference point for debugging.)

Shrinking must take realistic costs into account. While decreasing the value by 1 might seem fine for something around 10, if the minimal failing value were -9,223,372,036,854,775,808 (the minimum for a 64-bit signed integer), a method that decreases by 1 might not finish even in a day. Therefore, to finish tests within a reasonable amount of time, it's necessary to place restrictions on the shrinking method. The following restrictions are common:

  • Providing a base value
  • Limiting the number of shrinks

A base value is the value at which shrinking terminates if no smaller failing value is found. Generation and testing are repeated in a way that approaches this base value. The base value depends on the generator, but generally, 0 or empty is used. For numbers, the value is decreased toward 0; for strings or collections, the length is reduced toward empty. If no error occurs even at 0 or empty, the last value that triggered an error becomes the minimum value.

The number of shrinks varies by library. For languages with fast execution speeds, setting a higher limit shouldn't be a problem. Usually, 50 to 100 trials provide sufficient precision. While increasing the number of trials allows for more detailed identification, it also increases the load, creating a tradeoff.

Number of Shrinks

The number of shrinks is not necessarily better if it's higher. Generating shrink values and executing tests takes time. If you could shrink indefinitely, you would be more likely to find an approximation as close to the minimum as possible, but it would take an infinite amount of time. Therefore, every library imposes a limit on the number of shrinks (although jqwik and kiri-check allow for unlimited shrinking until the base value is reached).

To find the minimum value efficiently, you should specify the range of data rather than relying on the number of shrinks. For example, in the FizzBuzz code mentioned earlier, the error value closest to 0 is 15. The maximum value generated by integer() is 9,223,372,036,854,775,807, but in reality, a 64-bit test isn't necessary for FizzBuzz. Considering the use case of FizzBuzz, there's no need to try such large values. A range of up to 100 would be more than enough.

Algorithm

In generating values toward a base value, the simplest algorithm is the binary search method. Starting from the value that caused the error, the value or length is repeatedly divided in half. It looks something like "256, 128, 64, 32, 16, 8, 4, 2, 1, 0." The advantages of binary search are that it's easy to implement and can reach the base value in a small number of trials.

However, in cases like the FizzBuzz example (multiples of 15) where errors are sporadic, a simple binary search might skip over the values, making it difficult to find the minimum. For example, if shrinking starts from 105, the values tried would be 52, 26, 13, 6, 3, 1, 0. Although the minimum error value is 15, a simple binary search would identify 105 as the minimum. While increasing the number of divisions can improve precision, in practice, a reasonable degree of accuracy doesn't seem to pose major issues. Looking into the implementations of other libraries, they are often surprisingly simple.

Are Random Values Effective?

If a simple binary search can skip potential values, you might wonder if adding a random element would help—for instance, adding a random value to the result of a binary search or dividing by a random number. Randomness might increase the likelihood of finding a minimum value that's hard to catch with binary search alone.

However, it seems preferable for the data generated during shrinking to follow certain rules. While the goal of a generator is to discover errors, the goal of shrinking is to identify the error, and since this is the stage where humans intervene for bug fixes, reproducibility is important. Even if you could generate the same data by fixing the seed, it might become a source of confusion for the developer.

Shrinking During Value Transformation

Most libraries allow you to transform (process) generated values using methods like map. If an error occurs with the transformed value, does the shrinking target the value before or after the transformation?

This choice depends on the library. When the transformed value is the target, a generator associated with that value's type is often used. However, the library itself cannot determine which value—before or after transformation—is more deeply related to the cause of the error.

Character Codes, Dates, and Time Zones

Unicode

Unicode support varies among libraries. The range of Unicode code points is 0 to 10FFFF, representing over 1.11 million characters. Currently, there are about 110,000 assigned characters. This range is so vast that simple random generation is highly unlikely to produce the expected characters. To address this, many libraries provide generators that produce specific sets of characters, such as alphanumeric ones. However, Hypothesis seems to be the only major library that supports Unicode categories (kiri-check also supports them).

If you were to implement a library, it would be a good idea to analyze the UnicodeData.txt file distributed through the Unicode Character Database and extract all code points for each category. Using AI can help you generate the analysis code instantly. Depending on the language, there might be libraries for manipulating UnicodeData, but few probably allow you to retrieve code points by category (as it is not a common requirement).

Categories are finely subdivided; for example, there are multiple categories just for the alphabet. Some cases might require specific combinations of code points, like line breaks. If you want to prepare a basic character set, it's easiest to imitate existing libraries like Swift's CharacterSet.

Dates and Time Zones

The handling of dates and time zones varies from library to library. Many libraries only support actual dates, local time, and UTC. Since the specifications for dates (plus times) and time zones are extremely complex, it might be better for individual users to implement them according to their needs rather than the library trying to handle everything. Hypothesis supports time zones and appears to be the strongest library regarding dates. kiri-check also supports time zones, following Hypothesis's lead.

For the shrinking base value, Hypothesis uses January 1, 2000. While the appropriate base value might differ depending on the requirements of the system being developed, this seems like a safe value.

Imaginary Dates

Imaginary dates refer to dates that do not exist. These include February 29th in a non-leap year, the 31st of a month with only 30 days, and non-existent times that occur during the start and end of Daylight Saving Time (DST).

For instance, in the United States, the DST period typically runs from the second Sunday in March to the first Sunday in November, with the transitions occurring at 2:00 AM. When 2:00 AM arrives on the start date, the clock is advanced by one hour to 3:00 AM. Conversely, at 2:00 AM on the end date, the clock is set back one hour to 1:00 AM. These one-hour periods become non-existent time. On the start day, the hour between 2:00 AM and 3:00 AM does not exist; on the end day, the hour between 1:00 AM and 2:00 AM does not exist (in the sense of the transition). Moreover, the existence and periods of DST vary by country and region.

The main libraries supporting imaginary dates seem to be Hypothesis and fast-check. kiri-check also supports them, though it does not support imaginary dates related to DST (as it seemed quite difficult).

Summary

Actually building a library reveals that the quality of the generated data is surprisingly unreliable. And for good reason: it only selects at most a few hundred pieces of data from a vast number of combinations. Even if all tests pass, it might just be that data easy to pass was chosen by chance, and critical bugs are being overlooked. Furthermore, since you are implementing the same logic as the test target, you might end up with the same bugs in both the target and the test. In such cases, the bugs are easily missed as long as the tests pass. How about it? Starting to feel a bit uneasy?

While reading books on property-based testing is beneficial, I think building your own library is a great way to deepen your understanding. Fortunately, no special knowledge is required. When it first runs, you'll likely be surprised by data that doesn't seem useful at all. It's a good opportunity to think about how to generate useful data, what useful data actually is, and various aspects of testing. Since it's not the kind of product that generates direct value, it might not be the most cost-effective endeavor, but...

I would be happy to receive a badge if you enjoyed this. I would also love to hear your thoughts after using kiri-check. It will go toward my coffee fund and motivation.

Discussion