Flaky tests (aka fragile, brittle or instable tests) are tests that fail or pass intermittently without any changes to the codebase. They are a common problem in software development and can be frustrating to deal with. However, it is important to be disciplined with flaky tests because they can lead to a number of issues.

Why it is important to deal with a flaky, fragile or instable test suite

It is important to be disciplined with flaky tests because fixing flaky tests is essential for ensuring a stable and reliable testing process. Eliminating flaky tests is not only important to prevent time waste, but is also crucial to improving developers’ confidence in the tests.

• Firstly, flaky tests can cause a loss of trust in the test suite. If tests are failing, it can be difficult to know whether a failure is due to a genuine issue or a flaky test. This can lead to developers ignoring test failures.
• Secondly, flaky tests can slow down the development process. If developers are spending time investigating false positives or ignoring test failures, it can take longer to identify and fix genuine issues. This can lead to slower development cycles, lower productivity and a slower time to market.

Rapid Test Results from https://imgs.xkcd.com/comics/rapid_test_results.png  under Creative Commons BY-ND 2.5

Why is flakiness extremely critical when moving to Continuous Delivery?

Keep in mind that the robustness of tests and the maintainability is a major factor for the productivity of a development team. With the move to Continuous Delivery the aim is to execute tests more often and potentially with every change. If your test suite of system tests fails in 20% of the changes, that slows you down enormously. And you need a lot of time to investigate upcoming issues. Therefore, if you choose your test strategy and a tool for system testing, the robustness of your test suite should be one of the most important criterias.

The Top 5 Strategies for Avoiding Flaky Tests

So what are good practices for avoiding, identifying and reducing flakiness.

Best Practices for Writing Code and Tests to Reduce Flakiness

In this chapter, we’ll explore some best practices for writing tests that are reliable, deterministic, and less prone to flakiness. By following these best practices, you can ensure that your tests are testing the right things, catching real-world issues, and providing valuable feedback to your development team.

• Balanced Test Strategy: Have a lot more stable unit, component and isolated integration tests than system or scenario tests. Probably the most important to avoid flakiness.
• Test Isolation: Test Isolation is a essential technique for keeping lower level tests effective, maintainable, robust and fast. Testing with the real implementation can have the advantage to find more bugs and avoid that test code gets out of sync with the real implementation. Based on the architecture and design of the code you need to decide where to isolate and where to use the real implementation. E.g. it does not make sense to isolate from value objects. Test isolation is especially important to isolate from slow, fragile, expensive dependencies or dependencies, which would make the setup of the tests extremely complicated. Besides, it helps to simulate behaviour, which otherwise would be hard to test or verify. The Clean Code books and the Open Source repository contain a lot of guidance on how to apply test isolation and write stable tests. You can find more details in the blog Clean Code: Writing maintainable, readable and testable code
• Testable Architecture: There are also some basic patterns you can consider when defining your architecture.

How to Identify and Fix Flaky Tests

By identifying and fixing flaky tests, developers can ensure that the test suite is reliable and trustworthy. This can lead to faster development cycles, a quicker time to market, and a more robust codebase. There are several strategies, which can help with reducing flakiness.

• Keep the mainline running: Keeping the mainline running is a very important principle for reducing flakiness and staying productive because it helps ensure that the test suite is always in a working state. There is an agreement in the team that everyone first ensures that the test suite is running and not flaky before moving to the next task.
• Putting Flaky Tests under Quarantine: In the blog Flaky Tests at Google and How We Mitigate Them engineers from Google describe how they deal with flakiness.

Conclusion

Fragile Tests should be avoided, made stable immediately or put into quarantine.

How to Get Updates

I am regularly writing on different architecture and software engineering topics. You can follow me on different channels via following me on the SAP community, Twitter, LinkedIn or via substack (Views are my own).