Integration tests reach where unit tests can not
Unit tests – defined by their small scope, not always a single class – miss by definition often a realistic context. They work on that small scope and search for boundaries, special cases, and other scenarios that are related to the small scope. Because of this, they are very fast, simple in setup and come in great numbers.
The moment they interact with any external resource, as simple as an environment variable, as complex as a database system, they become integration tests. Unit tests can therefore not test things specifically to those external resources. Testing database indexes, for example.
The unit/integration test suites are one of the most common and known complementing type of tests.
Property-based tests verify what example-based test cannot write
Humans think in examples. The way we talk about is in examples. It is a go-to move when explaining complex problems. That is also the reason why example-based tests are popular. They describe a specific input and verify a specific output. This kind of tests are often easy to comprehend. Both when reading the tests are reading the test failure.
Boundaries and special cases are often forgotten with these examples. Writing all these scenarios in a maintainable and readable manner is also almost impossible. Property-based tests lets testers think on a higher level, where only the relationship between input/output is described. By the generative and randomized nature of these tests, spotting edge-cases is easier and provides more confidence in whether a piece of code works as expected.
Important to note that one approach does not eliminate the other, they almost cannot live without the other. The example-based tests are there for humans to read, property-based tests are there to flesh out the correct implementation. In case of a failure or missing scenario, the generative input can be adapted and a new example can be added. I like this approach especially to introduce new/junior people, without overwhelming them with the complexity of test properties.
Structural tests fill the gaps domain tests do not concern themselves with
Domain testing works with specifications of requirements. Those requirements – in their initial form – do not always contain the all the possible combinations of conditions. Code coverage tools could help identify missed conditional paths that were not fully traversed. In case of highly complex conditions, picking the most important ones based on the code coverage results is considered structural tests. They are more related to the source code itself than the domain problem they are trying to solve. These kind of tests are also possibly written to verify fallback scenarios that provide clean states in which the domain code can run.
Mutation tests verify what property-based tests should cover
Setting up the right test properties can be tricky. To know that all is covered and that future invalid implementations are caught, a ‘devil’s advocate’ approach can be used by muting tests. They validate the exists test suites are indeed verifying the expected outcomes. Returning `null`, `>=` instead of `>`, arithmetic changes are all examples of forcing the source code in an invalid state and proving the effectiveness of the test suites. In case of missed cases, the generative data in the test properties needs an update or the spread of values should be reevaluated.
Conclusion
Using only a single testing approach to verify all functionality of the implementation is a lot more riskier. Bugs come in all flavors and cannot all be caught with the same technique. Complementing tests that work together on different levels means that each suite focuses on specific context and specific kind of bugs. It simplifies test development and is the most safe approach on catching as much bugs as possible.
Domain in unit tests (examples & properties), infrastructure relationship in integration tests, structural tests for any missing link and fallbacks, mutation tests as cross-wide validation of test accuracy.
Thanks for reading!
Stijn
Subscribe to our RSS feed