Coexisting with Waterfall (Part 1): Unraveling the Complexity

What is this?

Reflecting on this past year while looking for a topic for an Advent Calendar, I came across this post.

At our company, we provide 4RAP, a product for financial institutions.
So far, we have implemented it for multiple financial institutions, such as SBI Securities and Ehime Bank, and in both cases, development for these financial institutions adopted the Waterfall model.
I believe both projects were completed relatively stably for their scale, avoiding schedule delays and large-scale failures.

Generally, it is considered difficult to lead a Waterfall project to success.
In fact, in the history of software development, exploratory development processes—including Agile—have been proposed, and through the accumulation of knowledge and improvements from practice, they have gradually been replacing Waterfall.
Frankly speaking, it is more rational to adopt exploratory methods if the conditions are met.
However, isn't it the reality that many still adopt (or are forced to adopt) Waterfall for various reasons?
For example, since 4RAP is a product that requires data exchange with financial institutions' systems, its implementation must align with the delivery timing of the systems it integrates with.
In such projects, exploratory methods—where the completion date cannot be explicitly stated—cannot be used, and Waterfall inevitably becomes the only option.

Many of you might say, "We don't do Waterfall."
But aren't there many situations where a project with a deadline comes down from the top... for various reasons?
Deadline-driven projects require planning and prediction to control their completion timing, and in many cases, their essence is ultimately no different from Waterfall.
Thinking about it that way, I suspect there are actually surprisingly many people who have opportunities to deal with Waterfall.
On the other hand, there seem to be very few articles mentioning Waterfall (only 7 results when searching on Zenn).
Given this situation, I thought there might be some social significance in sharing the knowledge I've gained so far, so I decided to write this article.

What I want to solve with this article

This article assumes "teams or companies that currently adopt or are forced to adopt Waterfall" as the audience. I would like to introduce my personal reflections and measures regarding why Waterfall doesn't work well and what is important to make it work.
I hope this article will be useful for readers facing challenges with failures in adopting the Waterfall model or controlling uncertainty, or for those looking for ideas to improve their development processes.

What is Waterfall?

For the sake of discussion in this article, I will redefine Waterfall as a series of processes involved in system development that satisfy the following conditions.

1. Presence of constraints that are difficult to change

The existence of strong constraints that are difficult to change, such as deadlines and budgets, is an important characteristic that defines a development process as Waterfall.
These constraints create a strong motivation to plan ahead to predict whether problems can be solved, which often leads to an incentive not to adopt exploratory processes like Agile.

2. Necessity to commit to requirements and specifications early in the period

The second characteristic of Waterfall is that a commitment, such as finalizing requirements and specifications, is required early in the period.
If there were merely constraints, there might be cases where exploratory methods are chosen, but when an early commitment is required, it becomes difficult to adopt exploratory methods due to time constraints.

3. Verification (testing) of requirements and specifications is conducted at the end of the period

The final characteristic of Waterfall is that the verification of whether requirements and specifications are met occurs only at the end of the development period.
Exploratory methods require a feedback loop to measure the divergence between the current location and the destination.
If only the first two conditions were met, applying exploratory methods might still be possible, though difficult.

However, in situations where verification only takes place at the end due to reasons such as resource or organizational deficiencies, the feedback loop does not function (or functions only in a limited way), and even if exploratory methodologies are introduced, they become a mere formality.
In other words, the nature of Waterfall as a non-exploratory = planned methodology is considered to be determined by this characteristic.

Why is Waterfall difficult?

In the previous section, we defined Waterfall. Now, let's delve into why Waterfall is difficult.

What is a "correct" system?

To understand the complexity of Waterfall, let's first focus on the concept of "correctness" in software development. What is important here are two dimensions: L-truth (Logic-truth) and F-truth (Fact-truth).
L-truth refers to logical correctness, meaning the software is technically correct—for example, it can be compiled without problems.
F-truth refers to factual correctness, meaning the software appropriately meets actual business requirements.
These concepts are used in TM (Theory of Modeling) by Masami Sato, where it is generally stated that a data model (and the system resulting from it) must satisfy these as binary true/false values.

For more on Masami Sato's TM, please refer to this article or the book.

However, in general software development, these are not simple binary values but have more complex gradations.

For example, in 4RAP, we adopt batch pipeline processing instead of stream processing with higher processing performance. This is based on the judgment that, given the current number of operators, batch pipelines can provide sufficient performance to satisfy the necessary L-truth.

For the background on why we adopted batch pipelines, please also refer to Recommended batch processing design for Effectively Once.

Similarly, from the perspective of F-truth, the data entry work associated with launching investment services such as new fund wraps or robo-advisors is another noteworthy example.
4RAP is a platform that supports the simultaneous operation of multiple investment services, but the frequency of service launches is not necessarily high.
Ideally, the lifecycle of an investment service should be controlled by the adopting financial institution. However, since development for such infrequent tasks does not match the expected returns, we made a compromise to allow for manual engineering intervention.

In this way, L-truth and F-truth are not simple binary values but have a wider range of gradations.

Therefore, in this article, I will redefine them as follows:

• L-truth: Non-functional requirements for the system
• F-truth: Functional requirements for the system
• The degree to which the system satisfies L-truth is "L-truth value L (0<=L<=1)", and the degree to which it satisfies F-truth is "F-truth value F (0<=F<=1)".
• When L=1 and F=1, the system is in an ideal state.
• The system has acceptable thresholds for L-truth and F-truth values, L* and F*.
• L* and F* are unknown.

Through this redefinition, I would like to delve deeper into the complexity of Waterfall and the concepts behind it.

What is the Complexity of Waterfall?

Based on the redefined L-truth and F-truth, a system can be said to exist in one of the following four areas:

• Area A (L < L*, F < F*): Satisfies neither functional nor non-functional requirements.
• Area B (L >= L*, F < F*): Satisfies non-functional requirements but not functional requirements.
• Area C (L < L*, F >= F*): Satisfies functional requirements but not non-functional requirements.
• Area D (L >= L*, F >= F*): Satisfies both functional and non-functional requirements.

Area A includes systems that are in the middle of development, as well as systems that are completed but insufficient for production use—where they don't even run properly, let alone support business operations. We occasionally hear about large-scale projects that were canceled; I believe those are cases where they couldn't escape Area A (how terrifying).
Area B corresponds to systems that work but lack the necessary functions to conduct business.
Area C refers to systems that meet functional requirements but cannot withstand production use, such as mockups or demos.
Area D is exactly the system we want to build—one that satisfies both functional and non-functional requirements.

The development process can be described as the procedure to manifest a system that belongs to Area D.

What, then, is the complexity of Waterfall as a development process?
For comparison, let's first consider Agile, one of the exploratory methods.
As stated in the Manifesto for Agile Software Development, a key characteristic of Agile is to continue providing "working software."

Manifesto for Agile Software Development

For example, if a customer asks for a "car as a means of transportation," this means first providing simpler means of transportation (items that satisfy L*) like a skateboard, a bicycle, a motorcycle, or a convertible.

By providing these various means of transportation, you verify the deficiencies in functional requirements and aim to reach the necessary and sufficient means of transportation that meets the functional requirements (in this case, a convertible).

Cited from Making sense of MVP

This represents conducting exploratory activities starting from Area B and aiming for Area D.

On the other hand, Waterfall is a method that requires a-priori commitment.
This represents aiming for Area D from Area C.

Both development processes are the same in that they aim to manifest a system belonging to Area D.
So, where does the cause of the complexity of Waterfall lie?

In the history of software development, a mathematical and scientific background has matured, and various software, frameworks, and cloud services have been developed to support development.
By using these support tools, creating a system that satisfies L-truth has become extremely easy.
This means that creating a system in Area B—where L-truth (non-functional requirements) is satisfied—is relatively easy, with little uncertainty.

On the other hand, F-truth (functional requirements) can change depending on contexts such as industry, company, and law.
For example, the Act on the Protection of Personal Information is revised every three years, and (naturally) the content of those revisions differs every time.
Furthermore, even with the same revision, the response will differ between a financial institution and a local shop's e-commerce site, and variations occur even among companies within the same financial sector.

Unlike L-truth, which has a historical accumulation, clarifying (predicting) the functional requirements F* that satisfy F-truth is a relatively difficult task because it requires understanding the specific context.

Furthermore, even if functional requirements are clarified (predicted), deriving a system that simultaneously satisfies L-truth is not self-evident.
Below is (part of) the famous image "What the customer really needed."
This picture satisfies the functional requirement of a "Tree Swing," but probably no one can ride it (and the tree will eventually wither), so it can be said that it does not satisfy non-functional requirements (production operation is impossible).

And as a result of exploring a working system (= satisfying L-truth), the functional requirement of a "swing" somehow completely fell through...
Whether it is this extreme or not, isn't this a common story?

The image "What the customer really needed" is cited from here

In other words, aiming for Area D from Area C in Waterfall involves two risks:

1. The commitment falls short of the true functional requirements.
2. The resulting system fails to meet the commitment as a result of exploring L-truth.

This is considered the cause of the difficulty in leading Waterfall to a successful conclusion.

Summary

In this first part, we covered the following key concepts and points to deeply understand the complexity of the Waterfall development process:

1. Concepts of L-truth and F-truth: To understand the challenges of Waterfall, we introduced the two dimensions of L-truth (logical correctness) and F-truth (factual correctness). These mean that the software is technically valid while simultaneously meeting actual business requirements.
2. Application of L-truth and F-truth in the development process: In actual software development, L-truth and F-truth are not simple binary values but have complex gradations. We extended and redefined the concepts of L-truth and F-truth to allow for the expression of these gradations.
3. Exploration of Waterfall's complexity: We redefined the development process based on the extended L-truth and F-truth and analyzed the complexity of the development process in Waterfall. Specifically, we showed that the difficulty of simultaneously satisfying functional and non-functional requirements and the uncertainty of prediction are the fundamental causes of Waterfall's complexity.

In this first part, we laid the foundation for understanding the essential challenges and complexities of Waterfall development. In the second part, we plan to delve deeper into how to deal with these challenges and further compare Waterfall with other development methods.