Key Differences Between Smoke and Sanity Testing

Some time ago, university software engineering teachers showed the traditional test phases using the waterfall process: the sequential line of unit test (UT), functional verification test (FVT), system verification test (SVT), and global verification test (GVT) to check the user-readable text. More enthusiastic teachers also showed black box and white box testing for test cases with and without access to the source code, respectively.

Over time, modern software development techniques emerged based on iterations, cycling feedback, and continuous integration. As a result, traditional testing phases became outdated and deficient. Surprisingly, we have developed processes focused on testing itself in modern days.

Software testing evolved into a more modern and accurate discipline. I see most of them as subroutines derived from traditional test phases. For instance, we have regression testing, which spawned two specialized subroutines: smoke and sanity testing.

What Is Smoke Testing? 

Smoke testing is the quality assurance process created to determine if the product build is stable. Once we have a successful result from smoke testing, the QA team can begin further test processes. In short, we evaluate the build’s quality before spending more resources on it.

Smoke testing is integrated by a minimal set of test cases required to guarantee the core functionality is working as expected. Sometimes smoke testing is referred to as “Build Verification Testing” or “Confidence Testing”.

The main point of smoke testing is to detect significant issues which could become “stoppers” later in the process. These stoppers can prevent us from delivering the build on time. Smoke tests are not designed to demonstrate system stability or alignment to new requirements. They are not “edge values”. Performance, integration, and other specialized tests are not intended to be executed. Under smoke testing, we follow the “happy path”. 

Test Automation in Smoke Testing

Regression testing is well suited for automation. Since smoke testing itself is a subset of the cases from regression testing, it also automates well. Automated smoke testing allows developers to check the build immediately. There are integration tools like Circle CI and GitHub Actions, which execute a smoke test as soon as the code is delivered to the hub.

Dev teams may have several developers integrating code multiple times in a single iteration, why wait for QA availability to run smoke testing? Developers getting feedback from testing in just a few minutes is a dream. Having automated smoke testing also frees testers from running boring, repetitive test cases. Instead, testers can spend time creating more automation or new elaborate cases.

What is Sanity Testing?

Like smoke testing, sanity testing is another type of regression testing. Both share the objective to verify that a code change in the software does not impact the existing functionality of the product. However, sanity testing ensures that the product works appropriately after new functionality and fixes are integrated into the build. Some test cases executed in past phases are re-executed to verify the influence of the changes. 

Sanity testing aims to check if development has solved the problems they set out to solve without creating new problems with their changes. So, sanity test cases need to be planned based on the lines of code touched in the build.

What’s the Difference Between Smoke and Sanity Testing? What Are They For?

A simple rule to differentiate between smoke and sanity testing is that smoke is intended to cover well-known functionality, such as the core functions of the product. Sanity testing is intended to do a simple test for new functionality, including fixes, in the current build.

As you see, both smoke and sanity testing check that no changes broke the code, but sanity testing is closely related to the modules or functions involved in the changes. So smoke testing could be the same set of test cases all the time for all the builds, no matter what was fixed, at least until new core functionality is created.

Did you catch that? Sanity testing is a fast way to test new functionality. Still, if that functionality becomes a primary function on the application, a subset of sanity test cases will be part of the smoke suite of test cases.

To end, if you need to quickly test a new feature before major testing, use sanity testing. Smoke testing should be executed recurrently as part of the build check. Sanity tests examine the new functionality, while smoke tests examine general functions at the application or system level. If testing were to be performed by a doctor, smoke testing is a general checkup, and sanity testing is a follow-up appointment for a previous injury.

Antonio Napoles