Trying Property-Based Testing on an Electric Kettle (5): Hunting for Unknown Issues

Introduction

Continuing from Trying Property-based testing on the "Wadai Futto Pot" (4) Boiling/Keep Warm - Changing implementation according to state transitions,
I have been exploring Property-based testing using fast-check.
As the subject matter, I decided to use the "Wadai Futto Pot" Requirements Specification (Model GOMA-1015) 7th Edition, which is specified as the test base for the U-30 Class of the Test Design Contest.
It seems that fast-check has an implementation for finding failing test cases by performing a random walk through states (Model-based testing), so today I would finally like to try that out.

What is Model-based testing?

Model-based testing here follows this flow:

1. To create a model, define the actions the test target is expected to perform and its states.
2. Compare and verify the actions of the test target against the results predicted by the model.

Model-based testing in fast-check

Model Based Testing Tutorial: What is, Tools & Example mentions that there are various types of models, such as:

• Data flow
• Control flow
• Dependency graphs
• Decision tables
• State transition machines

In fast-check, Model-based testing aims to test by creating a model of the state transitions among these.

State Transition Diagram

Let's look at the state transition diagram I considered in Trying Property-based testing on the "Wadai Futto Pot" (3) Boiling/Keep Warm - Thinking about state transitions.

It looks complex, but essentially I want to consider the following 6 states:

OFF_CLOSE = -1,
OFF_OPEN, //0
ON_IDLE, //1
ON_OPEN, //2
ON_ACTIVE_BOIL, //3
ON_ACTIVE_KEEP, //4

And the following actions:

• open(): Open the lid
• close(): Close the lid
• fill(): Add water
• dispense(): Dispense hot water
• reboil(): Reboil
• boil to keep: Wait until it boils

Creating the Model

Create the model. In fast-check, actions are defined as Commands. We define state, water, and temperature, which are visible as public members in Goma1015, and initialize them in the constructor just like in Goma1015.

import fc from 'fast-check'

import { Goma1015, State } from '../src/lib/index'

/** Class Goma1015Model */
export class Goma1015Model {
  /** to manage state transition */
  public state: number
  /** to manage water volume */
  public water: number
  /** to manage temperature */
  public temperature: number
  constructor() {
    this.state = State.OFF_CLOSE
    this.water = 0
    this.temperature = 25
  }
}

export type Goma1015Command = fc.Command<Goma1015Model, Goma1015>

Creating Actions

Basically, I want to find unknown problems, so I'll make each action executable in every state.
I will compare each model state with the actual instance.

open(): Open the lid. Operation condition: Starts in all states

The confirmation points are as follows:

• Does it transition to OFF_OPEN when open() is called in OFF_CLOSE/OFF_OPEN?
• Does it transition to ON_OPEN when open() is called in any other state?
• Is there no change in the amount of water inside?
• Does the water temperature become 25 degrees when the lid is opened, and is it reflected correctly?

Source

close(): Close the lid. Operation condition: Starts in all states

The confirmation points are as follows:

• Does it transition to OFF_CLOSE when close() is called in the OFF_OPEN state?
• Does it transition to ON_IDLE or ON_ACTIVE_BOIL when close() is called in the ON_OPEN state?
  • Does it transition to ON_ACTIVE_BOIL if the water volume is 10ml or more?
    • Is the temperature 25 degrees or higher after 1 second?
  • Does it transition to ON_IDLE if the water volume is 10ml or less?
• Is there no change in the amount of water inside?
• Does the water temperature become 25 degrees when the lid is opened, and is it reflected correctly?

Source

fill(): Add water. Operation condition: Starts in all states

The confirmation points are as follows:

• Does an Error trigger in any state other than OFF_OPEN or ON_OPEN?
• Does an Error trigger when overflowing the water?
• Other than the above, does the water accumulate properly in the pot when adding water in the OFF_OPEN or ON_OPEN state?

Source

dispense(): Press the dispense button. Operation condition: Starts in all states

The confirmation points are as follows:

• Does an Error trigger in any state other than ON_IDLE or ON_ACTIVE_KEEP?
• When dispensing in ON_ACTIVE_KEEP, if the water volume becomes 10ml or less, does it transition to ON_IDLE?
• Other than the above, when the dispense button is pressed in a state other than ON_IDLE or ON_ACTIVE_KEEP, does the hot/cold water come out in the specified amount from the pot?

Source

reboil(): Press the reboil button. Operation condition: Starts in all states

The confirmation points are as follows:

• Does an Error trigger in any state other than ON_ACTIVE_KEEP?
• Does it transition to ON_ACTIVE_BOIL when the reboil button is pressed in the ON_ACTIVE_KEEP state?
• Is there no change in the amount of water inside?

Source

boil to keep: Wait until it boils. Operation condition: Starts only during the boiling state (since it's not a user input)

The confirmation points are as follows:

• Is the temperature 25 degrees or higher after 1 second?
• Does it transition to ON_ACTIVE_KEEP when the water temperature reaches 100 degrees after sufficient time has passed?
• Is there no change in the amount of water inside?

Source

Execution

It seems the actual state transitions are being tried like this.

Wow, so many different states are being tested!

I ran 10,000 state transitions and they all passed!!

Conclusion

In an actual Wadai Futto Pot, there are timers, multiple keep-warm functions, and the states are much more complex. However, this time I built it step-by-step from the bottom up as much as possible.
In the future, I would like to add a UI or include the functions mentioned above. For now, I am satisfied that I could try out model-based testing with fast-check as intended.
I will conclude this series for now.
Also, if there are any mistakes in the code, Pull requests are very welcome.
Thank you very much for reading through all 5 articles in this series.