Property-Based Testing with TypeScript & fast-check

Originally published at here .

Have you ever heard about Property-Based Testing (PBT)? Before diving into PBT, let’s take a moment to recall our usual approach to testing.

Overview: What is PBT? Why PBT?

When developers write test code, they typically think of specific input values and the expected outputs. They write these examples as tests and run them on a computer. This approach, called example-based testing, is very common among developers. While this method can find bugs that the developer has predicted, it often misses unknown bugs.

That’s where Property-Based Testing comes in. PBT is a great way to find unexpected bugs early, leading to fewer issues in production. So, what exactly is Property-Based Testing?

What is Property-Based Testing?

Property-Based Testing generates hundreds or even thousands of test inputs at random and checks if certain properties are always true for the code being tested. Technically, the randomness isn’t completely random – it’s controlled and reproducible. This means that if a test fails, you can reproduce the exact scenario that caused the failure.

Example: Sorting an Array

Let’s start with a simple example. Suppose you have an array that includes some integer values, and you want to test a function that sorts this array.

Example-Based Testing

In example-based testing, you might write tests with specific arrays, like [1, 2, 3] or [4, 5, 6]. But what happens if the array is empty? Or what if it contains negative numbers? Writing tests for all possible cases can be difficult and time-consuming.

Here’s how you might write example-based tests for a sorting function using Vitest:

import { describe, it, expect } from "vitest";

// Function to test
const sortArray = (arr: number[]): number[] => arr.sort((a, b) => a - b);

// Example-based tests
describe("sortArray", () => {
  it("should sort an array of positive numbers", () => {
    expect(sortArray([3, 1, 2])).toEqual([1, 2, 3]);
  });

  it("should handle an empty array", () => {
    expect(sortArray([])).toEqual([]);
  });

  it("should sort an array with negative numbers", () => {
    expect(sortArray([3, -1, 2])).toEqual([-1, 2, 3]);
  });
});

Property-Based Testing

With Property-Based Testing, you define properties that should always be true no matter what the specific values in the array are. For example, you might say that sorting the array should always result in a sorted array. Then, PBT will generate many different arrays to test this property, including edge cases you might not have thought of.

Here’s how you can do this using TypeScript and fast-check:

import { test, expect } from "vitest";
import fc from "fast-check";

// Function to test
const sortArray = (arr: number[]): number[] => arr.sort((a, b) => a - b);

// Property-based test
test("Array is sorted by ascending order", () => {
  fc.assert(
    // fast-check generates the input for the test
    fc.property(fc.array(fc.integer()), (arr) => {
      const sortedArray = sortArray(arr);
      for (let i = 1; i < arr.length; ++i) {
        expect(sortedArray[i - 1]).toBeLessThanOrEqual(sortedArray[i]);
      }
    })
  );
});

Stateless and Stateful Tests in PBT

Now that we have a basic understanding of PBT, let’s look at two main types of tests: stateless and stateful.

Stateless Tests

Stateless tests are the simpler form of Property-Based Testing. They test functions or methods where the output depends only on the input values. Our sorting example is a perfect illustration of a stateless test. The property we’re testing is that the output array is always sorted, and this property doesn’t depend on any previous operations or states.

Stateful Tests

Stateful tests, on the other hand, are used to test systems that keep some form of state across multiple operations. For example, think about a banking system where you can deposit and withdraw money. The state of the system (the account balance) changes with each operation.

With stateful tests, you can generate sequences of operations (like a series of deposits and withdrawals) and check that the final state meets the expected properties (for example, the balance should never be negative). Stateful tests are especially useful for testing more complex systems where the sequence of operations can lead to different states, and making sure the system behaves correctly in all possible states is important.

Use Case: Vending Machine

To explain stateful testing, let’s think about a simple state machine like a vending machine. You can write tests that simulate sequences of operations like inserting coins, selecting products, and getting products. PBT will help you check that the machine always returns the correct change, gives the right product, and never enters an invalid state.

Here’s an example using TypeScript and fast-check for a vending machine:

Implementation of VendingMachine class

// Implementation
type Product = {
  name: string;
  price: number;
};

type VendingMachineState = {
  balance: number;
  products: Product[];
};

class VendingMachine {
  private state: VendingMachineState;

  constructor(initialState: VendingMachineState) {
    this.state = initialState;
  }

  insertCoin(coin: number): void {
    if (coin <= 0) {
      throw new Error("Coin value must be positive");
    }
    this.state.balance += coin;
  }

  pressButton(productName: string): number {
    const product = this.state.products.find((p) => p.name === productName);

    if (!product) {
      throw new Error("Product not found");
    }
    if (product.price <= 0) {
      throw new Error("Product price must be positive");
    }
    if (product.price <= this.state.balance) {
      this.state.balance -= product.price;
      const change = this.state.balance;
      this.state.balance = 0; // Reset balance after dispensing product & change

      return change;
    }

    return 0;
  }

  getBalance(): number {
    return this.state.balance;
  }

  getProducts(): Product[] {
    return this.state.products;
  }
}

In fast-check, Model-based testing is good way to control stateful tests.

fast-check explains Model-based testing like below.

Model-based testing can also be referred to as Monkey testing to some extend. The basic concept is to put our system under stress by providing it with random inputs. With model-based testing, we compare our system to a highly simplified version of it: the model.

ref: https://fast-check.dev/docs/advanced/model-based-testing/

It means that model-based testing with fast-check involves using Commands to create random actions and comparing the system's behavior to a simplified model to ensure accuracy. By simulating random sequences of operations and transitions between states, you can verify that the system behaves as expected under different conditions. This helps ensure the system is robust and reliable.

import { describe, test, expect } from "vitest";
import fc from "fast-check";

// The Commands for Model-Based Testing
type Model = {
  balance: number;
  products: Product[];
};

class InsertCoinCommand implements fc.Command<Model, VendingMachine> {
  constructor(readonly coin: number) {}

  check(m: Readonly<Model>): boolean {
    return this.coin > 0;
  }

  run(m: Model, r: VendingMachine): void {
    r.insertCoin(this.coin);
    m.balance += this.coin;
  }
}

class PressButtonCommand implements fc.Command<Model, VendingMachine> {
  constructor(readonly productName: string) {}

  check(m: Readonly<Model>): boolean {
    return m.products.some((p) => p.name === this.productName);
  }

  run(m: Model, r: VendingMachine): void {
    const product = m.products.find((p) => p.name === this.productName);
    if (product && product.price <= m.balance) {
      const change = r.pressButton(this.productName);
      m.balance = change;
    }
  }
}

// Property-based test
describe("VendingMachine", () => {
  test("Model-based testing with various products and coins", () => {
    fc.assert(
      fc.property(
        fc.array(
          fc.record({ name: fc.string(), price: fc.integer({ min: 1 }) })
        ), // price >= 1
        fc.array(fc.integer()), // allow any integer for coin
        (products, coins) => {
          const initialState: Model = {
            balance: 0,
            products,
          };

          const commands = [
            ...coins.map((coin) => new InsertCoinCommand(coin)),
            ...products.map((product) => new PressButtonCommand(product.name)),
          ];

          const s = () => ({
            model: initialState,
            real: new VendingMachine(initialState),
          });
          fc.modelRun(s, commands);
        }
      ),
      { verbose: true }
    );
  });
});

Tradeoffs

While PBT offers a significant advantage in uncovering hidden bugs, it can also feel complex and challenging to implement effectively.

Pros:

• Finds Unexpected Bugs: PBT is effective at uncovering bugs that example-based testing may miss by generating a wide range of test inputs.
• Reproducibility: The randomness in PBT is controlled and reproducible, allowing you to replicate and debug any failures.
• Efficiency in Testing Edge Cases: PBT automatically tests edge cases and unusual scenarios that developers might not think of.
• Scalability: PBT can handle more complex systems and scenarios, especially with stateful tests.
• Automation: Reduces the need to manually write numerous test cases, saving time and effort.

Cons:

• Complexity: Setting up PBT, especially for stateful tests, can be more complex compared to example-based testing.
• Initial Learning Curve: Developers need to learn new concepts and tools associated with PBT, such as model-based testing and fast-check.
• Performance: Running thousands of tests can be time-consuming and computationally expensive.
• False Positives/Negatives: Poorly defined properties can lead to false positives or negatives, requiring careful design of properties.

Summary

In summary, Property-Based Testing offers a powerful way to improve your testing strategy by finding bugs that traditional example-based testing might miss. By generating a wide range of test inputs and focusing on properties that should always be true, PBT helps make sure your code is strong and reliable. Also, understanding and using both stateless and stateful tests in PBT can greatly improve the quality and reliability of your software.

Thank you for reading! I hope this introduction to Property-Based Testing has been helpful.

Happy testing!