Testing is essential for any organization before they can think of releasing their product into the market. There are several kinds of tests that have to be undertaken at various stages in the software development life cycle. Today we take up the discussion of two such crucial tests, both of which are associated with making software more robust and reliable in the early stages of the development: Sanity Testing and Smoke Testing.
There is a certain amount of confusion surrounding these two testing types, often being used interchangeably To comprehend the fundamental differences between these tests, their advantages, and when each should be performed, one must first understand what it means by Sanity testing and Smoke testing. Before we dig deeper into that, we need to familiarise ourselves with the term “Software Build”.
What is a software build?
Any software is built up of source code files, which can run into thousands of files depending on how complex the software is. These files need to be put together into an executable form that can be further shared with the software testing teams. Thus, software builds are nothing but executable files of the software that are created as part of the software development cycle, especially for testing purposes. Builds are handed over to the testers to test for specified functionalities. When the development team provides a new build, the Software Testing team validates it and guarantees that there are no severe issues.
Now that we have understood what a software build is all about, let us explore smoke testing and sanity testing at length.
Smoke testing
Smoke testing is also referred to as a build verification test, and it ensures that there are no showstopper defects in the build that will prohibit the testing team from thoroughly testing the application. Through smoke testing, the testing team confirms that the build is stable before moving on to a more extensive level of testing.
As the Quality Assurance teams receive a new build from the development team, the first task they undertake is to do smoke testing. However, smoke testing is carried out even when new modules are added to the current functionality.
Smoke testing looks at a program’s core features to make sure it’s ready for further advanced testing. It can be challenging for a QA team to do a full test on software that can’t execute basic functions. Hence, the importance of smoke testing.
This type of testing takes its name from the literal “smoke” testing that is done in the context of hardware testing, where tests are conducted on hardware devices to ensure that they don’t release smoke when the power is switched on.
Under smoke testing, if the major features that aren’t working or key flaws haven’t been fixed, testing is suspended so that no more time is wasted installing or testing the build. The programmer’s primary priority becomes resolving the issues identified in the process of smoke testing. Smoke testing is a subset of acceptance testing.
Smoke Testing is part of those test cases that cover the main functionality of a component or system, with the goal of ensuring that the most important functionalities of a program operate while ignoring finer details. Some of the capabilities tested during the smoke test phase often include access to the application, logging in with a set of users (admin, regular user)
Smoke Testing: When do we use it?
As new software functionalities are built and integrated with an existing build, smoke testing should be performed. It checks for the operationality of all the core functionalities of the build. In the process, the stability of the build is also assessed.
The development team deploys the build-in QA in this testing procedure. Testers perform tests on the build after determining subsets of test cases to be used. The application’s critical functionalities are tested by the QA team.
The main goal of these chosen test cases is to uncover errors in the build being evaluated. It is only when all the tests return positive results, can the QA team move on to conducting Functional Testing.
Failure on any of the tests requires that the system be returned to the development team for fixing bugs. We undertake Smoke Testing to ensure the stability of the build whenever there is a change.
A small example will make this clearer: Say we are testing an app that has a login window and the development team adds a new registration button, and hence, the build is deployed with the updated code. Upon the addition of this new button, we need to conduct a smoke test.
Smoke Testing: Who will be in charge?
QA engineers/QA leads execute Smoke Testing after the build is delivered to the QA environment. Whenever a new build is released, the QA team determines the application’s major functionality in order to do smoke testing. The QA team looks for showstoppers in the application being tested.
Smoke Testing: What is the purpose?
Smoke testing is extremely vital in the software development life cycle since it assures the system’s accuracy in the early stages. It can help to bring down test efforts and overall testing time.
After the build has been released to QA, smoke testing is performed. Smoke testing identifies the majority of errors during the early stages of software development.
Smoke testing, hence, makes detecting and correcting severe flaws much easier.
The QA team can use smoke testing to uncover faults in the application’s functioning that may have been introduced by the new code.
Smoke Testing: How do you conduct it?
Smoke testing is normally done manually, although it is possible to automate the process. It may differ from one organization to the other, as per requirements and resources.
● Manual Smoke testing
The norm is to do smoke testing manually. Its methods differ from one organization to the next. Smoke testing is done to confirm that important path navigation is as expected, and that the functionality is not hampered.
● Automated smoke testing
We can run smoke tests against a set of automated test scenarios. Developers may check builds quickly with the help of automation tests once a new build is ready for deployment.
Smoke Testing: An example.
Let’s look at a simple Gmail application as an example of how to test it using the smoke technique.
What are the functionalities important when it comes to Smoke Testing? Remember these should be the core feature of the Gmail app:
● Log in to the Gmail application
● Compose an email
● Send the email
Now you may ask, why are these functionalities crucial for Smoke Testing?
Say that you can log into the Gmail app, and you can compose the email but assume your email doesn’t get sent. Does it make sense to test other features like Drafts, deleted messages, Archives, and so on? No, not at all. If even the most fundamental operation, such as sending an email, isn’t working, there’s little use in testing any further features.
Smoke testing is primarily used to test the essential sections of an application rather than the entire program.
Smoke Testing: The Benefits
- The ultimate quality of the final product is enhanced since smoke testing is more likely to uncover both functional and architectural problems, as well as component-level design flaws.
- This type of test is associated with incremental (new build, smoke test, rebuild, etc.) error diagnosis, and corrections are simplified.
- Smoke testing is simple to carry out.
- It helps in the early detection of errors and hence, contributes to shifting left in the SDLC.
- Smoke testing reduces the risk of failure.
- This testing makes it easy to track progress.
- It reduces effort and saves money and time.
The test aims to also check for the stability of the build and thus, is a value-addition in terms of the overall software stability
Sanity testing
Sanity testing is performed after getting the software build to confirm that the code modifications are working as planned. This testing is a checkpoint to see if the build, with its new modules, is stable enough for further testing to continue or not. It helps in speedily determining the quality of software regressions. Sanity testing is a subset of regression testing, which means that it can be automated as well.
Sanity testing complements regression testing by ensuring that the program meets user requirements after changes have been made. Sanity testing can also be used to quickly assess the state of a release and plan the following steps accordingly.
Sanity Testing: When do we use it?
There are various scenarios when sanity testing would make sense. For example:
- When the testing team receives a build after several regressions and/or with minor changes to code, sanity testing is a must.
- It should also be undertaken when the build has been sent after the flaw/bug has been resolved.
- Right before the product deployment, sanity testing proves to be a good idea.
Sanity Testing: Who will be in charge?
Sanity testing is code testing done in a development environment to check the correctness of the program before delivering the build to QA. It is frequently done in a narrow and deep manner. It’s a procedure for ensuring that an application in development meets its fundamental functional requirements.
Sanity testing helps to determine whether the development process is complete and whether the software product should be passed or rejected for further testing.
Sanity Testing: What is the purpose?
The major goal of this testing is to ensure that the suggested improvements or functionality works as planned. If the sanity test fails, the build is rejected by the testing team, saving time and money. It’s only done once the build has passed the smoke test and been approved for additional testing by the Quality Assurance team. During this testing procedure, the team’s primary focus is on validating the application’s functionality rather than performing extensive testing.
Assume that an e-commerce project has primary modules: a login page, a home page, a user profile page, a new user creation page, and so on.
The login page has a bug where the password field permits less than four alphanumeric characters, despite the fact that the requirement states that the password field should not be less than eight characters. As a result, the testing team reports the bug to the development team for resolution. When the development team has fixed the problem and has passed it on to the testing team. The testing team then verifies that the changes made are final. It also checks to see if it has any effect on other connected functions.
On the user profile page, there is now a feature to update the password. We’ll need to validate the login and profile pages as part of the sanity check to make sure the tests are working properly in both places. Sanity tests are typically conducted for builds that require quick production deployment, such as a critical bug repair.
Sanity Testing: How do you conduct it?
Sanity testing, unlike other types of software testing, does not include a set of procedures. Because you want to perform quick and speedy testing, you don’t need to script the test cases for sanity testing. As a result, the initial step is to identify any new features, updates, or bug fixes.
You’ll now test the newly implemented changes to see if they’re operating properly. Then you’ll test a couple of similar features at random to determine if they’re also working as planned. If everything goes well, the release can be forwarded on to be thoroughly tested.
Sanity Testing: An Example.
Let’s have a look at an OLA application for testing as an example of sanity testing.
The basic features of the app are as follows:
● Sign up for the OLA app
● Log in to the app
● Search Cab
● Book Cab
Assume these four functionalities have been tried and the “Login” functionality is not working. This problem is then reported to the development team. The developer then adjusts and corrects the flaws.
Subsequently, the updated function will be tested for sanity. Here, the updated version of the feature should not interfere with the proper functioning of the other features.
Sanity Testing: The Benefits
- Speedy Evaluation.
First and foremost, sanity testing is quick. The functionalities or areas to be tested in sanity testing have a very tight focus. You don’t need to script the test cases before performing sanity testing; instead, you can utilize an unplanned, natural method. Since no documentation is required, sanity testing can be conducted in a speedy fashion. - Sanity testing helps you avoid wasting time and effort on unnecessary testing.
Sanity testing decides whether or not the application needs to be evaluated further. This saves time for testers if the release is in too bad of shape to pass rigorous testing. Additionally, testers are not required to record or log issues, which reduces reporting time. - Sanity testing is a subset of regression testing.
This means that it can help improve overall regression testing results. It is confirmed that the program is functioning and meets the change’s requirements, or that new functionality has been added. - Sanity testing can reveal any deployment or compilation problems.
For example, the tester may get a bad user experience if the developers did not generate the build utilizing all resource files. There could be any deployment notes that the development team overlooked. The release may not work, or even load, in the test environment as a result of the oversight. - Finally, sanity testing provides a brief snapshot of the product’s current status, allowing you to plan your next steps accordingly. If the sanity test fails, your development team may decide to postpone the next job and focus on fixing the flawed release first. If the sanity test is passed, you can ask your development team to move on to the next task while maintaining only one developer working on bug repairs or dedicating 1-2 hours per day for bug fixes.
Thus, we have seen in great detail what the meaning, purpose, and benefits are of both smoke testing and sanity testing. Let us now circle back to our main discussion at hand:
Smoke testing vs Sanity Testing: The real differences
- Smoke testing ensures “stability,” whereas sanity testing verifies “rationality.” Smoke testing is done by both developers and testers, whereas sanity testing is done by testers.
- Sanity Testing examines bug patches, whereas Smoke Testing looks at the system’s vital functions.
- Smoke testing is part of acceptance testing, while sanity testing is part of regression testing.
- Smoke testing can be documented and is, whereas sanity testing is neither documented nor scripted.
- The entire system is examined by smoke testing, whereas sanity testing only verifies a particular component of the system.
- Smoke testing is first done on the initial software build, while Sanity testing is conducted on stable builds or for newly added features.
Hence, we see that, while both are vital components of the overall testing system, they have some basic differences, as noted above.
Let’s quickly look at some other points that are important when discussing sanity and smoke testing:
● Sanity and Smoke tests both avoid wastage of time and effort by identifying whether an application is too defective at the early stage, to be tested thoroughly.
● A build verification test is smoke testing that is run on a specific build.
● Conducting a daily build and smoke test in software projects is one of the finest industry practices.
● Smoke and sanity tests can be done manually or with the help of an automation tool. When using automated tools, the tests are frequently started by the same process that creates the build.
You may need to run both Sanity and Smoke Tests in the software build depending on the testing requirements. In such circumstances, you will do Smoke tests first, followed by Sanity Testing. Sanity testing test cases are frequently paired with smoke test cases in the industry to speed up test performance. This is the reason why often the terms are confused and also sometimes used interchangeably.
The Conclusion: Smoke testing vs Sanity testing
Let’s end the conversation and review what we’ve talked about so far. The terms sanity testing and smoke testing are often used interchangeably in the industry. But both these are quite distinct.
Smoke testing is a type of testing undertaken to check whether the core functionalities of the software build are performing as expected or not, and this needs to be done before other detailed tests are conducted. Sanity testing is a technique for determining whether your software release is worthy of being thoroughly tested or is too faulty to be tested. It is performed when a new feature, a change request, or a bug repair is added to the code and the scope is narrow and focused on only functionalities which have been impacted by the change.
After discussing both testing types in-depth, we highlighted the variations that exist between sanity testing and smoke testing. While each has its role to play, both Sanity Testing and Smoke Testing are time and money-savers in the sense that they determine whether an app is too faulty to go for further testing.