On Testing Implementation Details

Introduction (Translator): I am not extremely proficient in English. Please read with a kind heart. Also, please refer to the original article as well.

This is a translation of the article Testing Implementation Details by Kent C. Dodds

Back when I was using enzyme (like everyone else at the time), I handled certain enzyme APIs with caution. I completely avoided shallow rendering, and I never even used instance(), state(), or find('ComponentName').

In code reviews for other people's pull requests, I've explained over and over again why these APIs should be avoided. It's because these APIs allow you to test the implementation details of a component. I've often been asked what "implementation details" means. I use it in the sense that it's something difficult to test for what it is. Why do we need to limit these APIs to avoid testing implementation details?

Why is testing implementation details bad?

There are two clear reasons why you shouldn't test implementation details. Tests that test implementation details:

1. May fail even though you only refactored the application code False negatives
2. May not fail when the application code is not working properly False positives

Testing is about "whether the software works." If a test passes, it means the test result is positive. If a test fails, it means the test result is negative. "False" refers to the test producing an incorrect result. In other words, it refers to cases where the test succeeds even though the software is actually broken (false positive), or the test fails even though the software is actually working (false negative).

Let's look at each reason in turn, using the simple accordion component below as an example.

// accordion.js
import * as React from 'react';
import AccordionContents from './accordion-contents';

class Accordion extends React.Component {
  state = { openIndex: 0 };
  setOpenIndex = (openIndex) => this.setState({ openIndex });
  render() {
    const { openIndex } = this.state;
    return (
      <div>
        {this.props.items.map((item, index) => (
          <>
            <button onClick={() => this.setOpenIndex(index)}>
              {item.title}
            </button>
            {index === openIndex ? (
              <AccordionContents>{item.contents}</AccordionContents>
            ) : null}
          </>
        ))}
      </div>
    );
  }
}

export default Accordion;

You might wonder why I'm using a class component instead of a function component (+ hooks) for the example. Don't stop reading yet! This is a sharp observation (though those who know enzyme might have already guessed why).

And here is the test that's testing implementation details:

// __tests__/accordion.enzyme.js
import * as React from 'react';
// If you're wondering why not use shallow
// read this: https://kcd.im/shallow
import Enzyme, { mount } from 'enzyme';
import EnzymeAdapter from 'enzyme-adapter-react-16';
import Accordion from '../accordion';

// set up the enzyme react adapter
Enzyme.configure({ adapter: new EnzymeAdapter() });

test('setOpenIndex sets the open index state properly', () => {
  const wrapper = mount(<Accordion items={[]} />);
  expect(wrapper.state('openIndex')).toBe(0);
  wrapper.instance().setOpenIndex(1);
  expect(wrapper.state('openIndex')).toBe(1);
});

test('Accordion renders AccordionContents with the item contents', () => {
  const hats = { title: 'Favorite Hats', contents: 'Fedoras are classy' };
  const footware = {
    title: 'Favorite Footware',
    contents: 'Flipflops are the best',
  };
  const wrapper = mount(<Accordion items={[hats, footware]} />);
  expect(wrapper.find('AccordionContents').props().children).toBe(
    hats.contents
  );
});

Raise your hand if you've ever seen (or written) a test like this in your own codebase (🙌)

OK, let's look at what problems these tests cause.

False negatives when refactoring

A surprising number of people find tests (especially UI tests) unpleasant. Why is that? There are various reasons, but one of the biggest reasons I've heard many times is that a lot of time is spent taking care of the tests. "Every time I change the code, the tests fail!" This is a huge drag on productivity! Let's see how our tests cause this frustrating problem.

Suppose you are refactoring the accordion to allow multiple accordion items to be open at once. Refactoring means changing only the implementation and not changing the existing behavior at all. So, let's change the implementation without changing the behavior.

We will change the internal state from openIndex to openIndexes so that multiple accordion elements can be open at once.

class Accordion extends React.Component {
-  state = {openIndex: 0}
-  setOpenIndex = openIndex => this.setState({openIndex})
+  state = {openIndexes: [0]}
+  setOpenIndex = openIndex => this.setState({openIndexes: [openIndex]})
  render() {
-    const {openIndex} = this.state
+    const {openIndexes} = this.state
    return (
      <div>
        {this.props.items.map((item, index) => (
          <>
            <button onClick={() => this.setOpenIndex(index)}>
              {item.title}
            </button>
-            {index === openIndex ? (
+            {openIndexes.includes(index) ? (
              <AccordionContents>{item.contents}</AccordionContents>
            ) : null}
          </>
        ))}
      </div>
    )
  }
}

You open the app, check the behavior, and confirm that everything is working properly. Great! Now, adding the feature to open multiple accordions to the component later should be easy! So let's test it. 💥 Boom 💥 The test is broken. Which one is failing? It's setOpenIndex sets the open index state properly.

The error message says:

expect(received).toBe(expected)

Expected value to be (using ===):
  0
Received:
  undefined

Does this test failure warn us about a significant problem? No! Right now, the component is working perfectly fine.

This is what is called a false negative. It means the test failed not because the application code was broken, but because the test was flawed. Honestly, I can't think of a more frustrating test failure than this. Well, let's pull ourselves together and fix the test.

test('setOpenIndex sets the open index state properly', () => {
  const wrapper = mount(<Accordion items={[]} />);
-  expect(wrapper.state('openIndex')).toEqual(0);
+  expect(wrapper.state('openIndexes')).toEqual([0]);
  wrapper.instance().setOpenIndex(1);
-  expect(wrapper.state('openIndex')).toEqual(1);
+  expect(wrapper.state('openIndexes')).toEqual([1]);
});

Conclusion: Tests that test implementation details may give you false negatives when refactoring. This can make tests feel fragile and frustrating, as if they break just by looking at the code.

False positives

Now, let's say a colleague is working on the accordion component and finds the following code.

<button onClick={() => this.setOpenIndex(index)}>{item.title}</button>

He might immediately feel like doing some premature optimization and think, "Alright! Inline arrow functions in render are not good for performance, so let's get rid of it! It'll probably work, so I'll just fix it quickly and test it."

<button onClick={this.setOpenIndex}>{item.title}</button>

Cool. When he runs the tests... ✅✅ Great! He committed without checking in the browser. Since the tests passed, it should be fine, right? The commit was mixed into a PR with thousands of lines of unrelated changes, and naturally, it was overlooked. In production, the accordion broke. Nancy can't see her tickets for the Wicked show in Salt Lake City next February (Translator's note: I'm not confident about the translation of "Nancy..."). Nancy is crying, and your team feels terrible.

What went wrong? We verified that state changes when setOpenIndex is called, and that the accordion content is displayed correctly, right!? Yes, exactly! However, the problem was that we hadn't tested whether the button was correctly wired up to setOpenIndex.

This is what is called a false positive. It means the test passed, but it should have actually failed! How can we prevent this from happening again? We need to add a test to verify if the state is correctly updated when the button is clicked. And to avoid repeating this mistake, we need to keep code coverage at 100% at all times. Oh, and we'd have to create a bunch of ESLint plugins to stop others from using Enzyme APIs that encourage testing these implementation details!

...But that's such a hassle... Ugh, I'm just fed up with false positives and false negatives. Maybe it's better not to write tests at all. Let's just delete all the tests!! If only there were a tool that let you write in best practices automatically... No, such a tool actually exists!

Tests that don't test implementation details

You could rewrite all your tests in Enzyme while limiting the APIs you use to avoid testing implementation details, but instead, I'm going to use React Testing Library, which makes it difficult to test implementation details in the first place. Let's take a look at React Testing Library!

// __tests__/accordion.rtl.js
import '@testing-library/jest-dom/extend-expect';
import * as React from 'react';
import { render, screen } from '@testing-library/react';
import userEvent from '@testing-library/user-event';
import Accordion from '../accordion';

test('can open accordion items to see the contents', () => {
  const hats = { title: 'Favorite Hats', contents: 'Fedoras are classy' };
  const footware = {
    title: 'Favorite Footware',
    contents: 'Flipflops are the best',
  };
  render(<Accordion items={[hats, footware]} />);

  expect(screen.getByText(hats.contents)).toBeInTheDocument();
  expect(screen.queryByText(footware.contents)).not.toBeInTheDocument();

  userEvent.click(screen.getByText(footware.title));

  expect(screen.getByText(footware.contents)).toBeInTheDocument();
  expect(screen.queryByText(hats.contents)).not.toBeInTheDocument();
});

Sweet! A single test successfully verifies all behaviors. Also, this test will pass regardless of whether the internal state name is openIndex, openIndexes, or tacosAreTasty 🌮. Nice! We've eliminated false negatives! Also, if the click handler wasn't wired up correctly, the test would fail. Sweet! We've eliminated false positives as well! And you don't even have to remember best practices about which APIs to avoid. Just by using React Testing Library normally, you get tests that give you confidence that the accordion works exactly as the user expects.

What is "Implementation Details" anyway?

Here is a very simple definition I came up with:

• Implementation details are things which users of your code will not typically use, see, or even know about.

Now, let's address the first question you're likely to have: "Who is the user of this code?" First, the end-user who interacts with the component in the browser is clearly a user. The end-user sees or touches the rendered buttons and content. However, developers are also users. Developers render the accordion with props. In summary, there are generally two types of users for a React component: end-users and developers. End-users and developers are the "users" your application code should consider.

Great. So, where are the parts in the code that users use, see, or know about? End-users interact with or see what is rendered by the render method. Developers interact with or see the props passed to the component. Therefore, generally, your tests should only deal with the passed props and the rendered results.

This is exactly what React Testing Library tests do. You pass an accordion component with fake props to React Testing Library, check for content as the user would see it (or verify it's not there), and interact with the rendered results by clicking the rendered buttons.

Think about the Enzyme tests. In Enzyme, we were referencing the openIndex state. Users don't care about this directly. They don't know what that state is named, they don't know if the open index is stored as a primitive or an array, and frankly, they don't care. They also don't know or care about the setOpenIndex method. However, the Enzyme test knows about these two implementation details.

This is why Enzyme tests are prone to false negatives. Because you're writing tests in a way that differs from how end-users or developers use the component, a third user of the application code emerges: the test! Frankly, the test is a user that nobody cares about. We don't want the application code to have to consider the test. It's a complete waste of time. We don't need tests written for the sake of the tests. Automated tests should verify that the application code is working correctly for production users.

The more your tests resemble the way your software is used, the more confidence they can give you.
— Me

For more details: Avoid the Test User.

About hooks

Enzyme still has many issues regarding hooks. As we've seen, testing implementation details significantly affects tests when the implementation changes. This is a ticking time bomb. For example, when migrating from a class component to a function component + hooks, your tests won't help you confirm if you've broken something during the process.

How about React Testing Library? It works great. Check out the CodeSandbox link at the end for the behavior. I love running tests written with React Testing Library.

Implementation detail free and refactor friendly.

Conclusion

Now, you know how to avoid testing implementation details, right? Using better tools is a good starting point. Let me tell you how to figure out what you should be testing. Following this approach will help you maintain the right mindset when testing and naturally avoid implementation details:

1. Does the untested part of the codebase lead to the worst-case scenario when it breaks? (Think about the app's checkout process)
2. Break the code into several units. (Clicking the "checkout" button sends a request to /checkout with the list of items in the cart)
3. Think about who the "user" is for that code. (A developer renders the checkout form, and the end-user clicks the button on the form)
4. Write a list of instructions to manually test that the code isn't broken. (Render the form with fake data in the cart, and verify that the mocked /checkout API is called with the correct data when the checkout button is pressed. Also, verify that a success message appears on the screen when the mocked API returns a successful response)
5. Create automated tests based on the list of instructions.

I hope these help! If you want to take your testing methodology to the next level, I highly recommend getting a Pro license for TestingJavaScript.com🏆.

Good luck!

P.S. If you want to check the code content in the article, there is a CodeSandbox here.

P.S.P.S. Here's a practice exercise for you: if you rename the AccordionContents component, what will happen in the second Enzyme test?