Test purpose, scope and location
A successful smoke test run does not mean that the software is correct, it only means that the software is not catching fire (read: completely broken). These type of tests are by their nature simple and should always be fast. Implicitly, the assertions in smoke tests are usually the things that integration/system tests build upon. A good example is a health endpoint for your application. The smoke test could be that this endpoint returns 200 OK, if the endpoint returns anything different, it would be pointless to run integration tests additionally which interact with other endpoints of the app.
Unlike the clear scope of unit or integration tests, smoke tests are a bit more ambiguous when defining their scope. That is because they are not bound by software or infrastructure, they talk about the most basic/critical ‘things’ that need to be there before we can talk about an ‘application’, without describing the application inner workings fully.
Each new build, deploy or even code change could benefit from a smoke test run. If there is already something of a ‘package’ at the PR level, like deploying to a test environment, it can even be part of the branch policy of the PR. Because they are cheap, they can (and should) be able to run at many places.
Seams between other test suites
‘See if an user can authenticate with an endpoint’ can be considered at first glance to be a smoke test by its simple setup and its lack of concrete use case. But on the other hand, if the software is authentication software, than it might be better suited as an acceptance test.
Smoke tests might be in some cases better suited to be incorporated within other test suites, because it might want to reuse a lot of code. If there are a lot of tests that build on top of ‘user is authenticated’, then those tests might be marked as ‘integration tests’ while the ‘can be authenticated’ tests can be marked as ‘smoke tests’ within the same test suite. With command options/flags (like dotnet test filter) the right portion of the test suite could be run in the right circumstances.
Same goes for the acceptance tests during deployments, which actually works with use cases instead of technical details. The essential features might be given priority before we release the new code to the next test phase.
The main benefit for having smoke tests run before other test suites is fast feedback. They could also be of use as a small check for development deployments where manual testing happens. The manual ‘clicking around the application’ could be written as a smoke test after each deployment. If those ones fail, there is something fundamentally wrong with the product.
Functional and technical
While there is a lot of resources and emphasis on software-related smoke tests, there is also a whole range of infrastructure-related opportunities. With the arrival of Bicep/ARM, the developer team has more responsibility on delivering correct infrastructure besides correct software.
Azure policies provides a way describe some of these technical checks on deployed resources, but the problem with this is that now the implementation (Bicep/ARM) is responsible to test itself. Yes, it can be a good way to force those policies on resources after deployment and verify against those, but by themselves cannot be considered a smoke test suite due to its false positive relationship. It should be the test suite that forces assertions on the implementation.
Technical infrastructure smoke tests could be run after each deployment and could be as simple as checking if resources use the right SKU and as critical as checking if only a certain set of role definitions are set in a resource group. The important thing to note here is that it is often just as valuable to test if something IS NOT there as it is to test if something IS there: if an user is assigned as an admin, but should only get reader access, asserting that the user has reader access is not enough; it should also assert on having no higher privileges.
The Pester PowerShell testing library might be a good way to describe these infrastructure tests as the typical Azure PowerShell commands can be used to interact with the deployed resources.
Conclusion
Smoke testing can be very valuable if used wisely. Having them fast means you can run them more frequently; having them broad, unburdened of details, means they hold their value longer. Just like any test suite, they should grow with the target they depend on, whether it be an integration test suite or a Bicep file. They are defined by their target – that is their whole reason to exist.
Also think broader than just test suites and infrastructure, think about what makes your application what it is. If you are developing a library and deploying to NuGet; it might beneficial for example to check if each project in the solution is indeed generating a NuGet package after the compile step. A simple check, but with a big impact when things start to change.
Find the smoke and you will find the fire.
Thanks for reading!
Stijn
Subscribe to our RSS feed