In today’s fast-paced modern world, test automation has become an essential part of the technology and software industry. These automated tests are used in place of manual tests, and there are tons of benefits to them! But that doesn’t mean that there still aren’t doubters and skeptics. In fact, a lot of individuals are unsure of automated tests. There are many companies that are convinced that automation testing is very expensive, not even realizing that they can be saving tons of money and effort. One of the best ways to help alleviate those doubts is to take a look at the Test Automation RoI, or return on investment. That way, companies can get an idea of how beneficial automated tests can be to their company.
So, what exactly is return on investment?
First, let’s ask ourselves, what is return on investment, and what does it show us? To put it simply, return on investment lets us know how successful our investments are. It’s a concise and easy metric that can help us see the efficiency of an investment into automated testing. It’s a wonderful tool for companies to use to estimate whether or not automated testing is right for them. Plus, it’s a great way to compare different test automation services. Basically, the return on investment lets companies know how much they’ll make (or lose) on their investment. It’s the same thing that everyone does with their own, everyday investments!
How do we calculate Test Automation RoI?
Having a positive return on investment is crucial to success for automated tests. But how does a company know if they’ll end up with a positive return? Well, the first thing that they need to establish is how much they would be investing in automated tests. This doesn’t just mean money. It also means resources like time and employees, figuring out the quality of the automation software, as well as the speed of the test execution. Calculating the return on investment has multiple benefits, not just monetary ones. For instance, it helps companies to compare diverse test automation services. It also helps them to test the quality of different automated tests. As one might guess, there’s a very simple equation to figure out the return on investment. The first thing one must do is subtract the cost of the investment from the value of the investment. And then, it’s the simple task of dividing the net gain from the total necessary investment.
So what do we need to calculate the return on investment?
As with all comparative equations, figuring out the return on investment requires framework. There are several different areas that must be explored to accurately come up with a concise return on investment.
One of the biggest mistakes one can make when calculating Test Automation RoI or return on investment for automated testing, is not calculating the time and effort it takes to work on the tests that are still manual. Yes, even with automation, there are tests that will be manual. This is something that is often overlooked during these calculations, and that can be a fatal error when calculating return on investment. It is absolutely crucial for companies to be succinct and thorough when doing these calculations.
Let us create a spreadsheet to help you calculate the RoI. As stated earlier, RoI can be summarized with the following simple equation:
RoI = return/cost * 100%
The above equation has just two components:
- Cost
- Return
Looks easy, right? But how do I calculate the cost and what is the return? Lets explore this further with an example.
Say, you have a cutting edge SaaS product with tons of users. You currently have about 300 manual regression tests that are run before every release. With that in mind, you need to start making some assumptions. These assumptions could be based on real time data if you keep track of it, or based on your discussions with the team.
- How many tests, at an average, do you add per release?
- How many existing tests do you modify? This could be due to a new field being added to an existing screen or a new feature touching existing functionality.
- How many tests get removed due to deprecated/removed functionality?
- Average effort for modifying existing and creating new tests.
- We all know not everything can be or rather should be automated. So what is the percentage of existing tests, that can not be automated. They will still be run manually.
- Does you application support multiple browsers/devices/platforms. In today’s world, the most likely answer is yes. So you also need to consider how many browsers/devices/environments would the tests be run on.
The table below, lists these assumptions. If you notice, we are using percentages for these assumptions as opposed to absolute numbers. This makes the calculating Test Automation RoI a bit more complex, but more realistic.
The table below lists the effort for creating and executing manual and automated tests.
With these assumptions in place, we are all set to calculate our Test Automation RoI.
Calculating Test Automation RoI
Let us first see what constitutes costs. In a simple scenarios where you already have manual tests, and you just need execute these manual tests, calculations is as simple as
Cost for manual test execution = Number of tests * time to execute each test in hours * cost per hour
Based on our assumptions above, this translates to
Cost for manual test execution = 300 * 2 hours per test * $20 per hour = $12000
For Automation tests, this is not so simple, we need to consider the cost to automate existing tests, and running some test manually since not all can be automated.
Total number of tests to be automated = 300 – tests that can not be automated = 300 – 10% = 270
Cost for creating new automated tests = 270 * 12 hours per test * $20 = $64,800
Cost for automated test execution = 270 * 0.1 hours per test * $20 per hour = $540
Cost for manually executing remaining test = 30 * 2 hours per test * $20 = $1200
Total Cost for Automation (Creation + Execution) = $64,800 + $540 + $1200 = $66,540
The above calculations clearly show that test automation has a very high upfront cost. If you test your app just once every release, then automating your tests may not be the right path. The table below, shows a detailed view of the calculations for 10 Releases, taking into account the growth of tests per release and their corresponding effort and cost.
Test Automation RoI after the 10th Release= ($179,235 – $214,565)/$214,565 = -16%
Test Automation Benefit or Test Automation RoI
Now, let us take a look at a little more realistic example for determining Test Automation RoI. Given the same set of assumptions, we just increase our target environments to 4. That is, you have discovered that your users mostly use Windows 10 and only 4 browsers = Chrome, Firefox, IE and Edge
Since we are executing the tests manually, we spend 4 times the effort since the same tests need to run manually on each environment.
Cost for manual test execution = 300 * 2 hours per test * 4 environments * $20 per hour = $48000
Cost for creating new automation tests remains the same as described in the previous section.
Cost for creating new automated tests = 270 * 12 hours per test * $20 = $64,800
Cost for automated test execution = 270 * 0.1 hours per test * 4 environments * $20 per hour = $2160
Cost for manually executing remaining test = 30 * 4 environments * 2 hours per test * $20 = $4800
Total Cost for Automation (Creation + Execution) = $64,800 + $540 + $1200 = $66,540
The differential between manual and automation is much less this time around. The table below, shows a detailed view of the calculations for 10 Releases, taking into account the growth of tests per release and their corresponding effort and cost. We can clearly see the benefits of automation in this scenario from the second release/sprint onwards.
Test Automation RoI after the 10th Release= ($654679 – $269309)/$269309 = 143%
Interpreting the Test Automation RoI
When interpreting ROI calculations, it’s important to keep a few things in mind. First, ROI is typically expressed as a percentage because it is intuitively easier to understand (as opposed to when expressed as a ratio). Second, the ROI calculation includes the net return in the numerator because returns from an investment can be either positive or negative.
When ROI calculations yield a positive figure, it means that net returns are in the black (because total returns exceed total costs). Alternatively, when ROI calculations yield a negative figure, it means that net returns are in the redbecause total costs exceed total returns. (In other words, this investment produces a loss.) Finally, to calculate ROI with the highest degree of accuracy, total returns and total costs should be considered.
As the graph below demonstrates, cost for automation grows linearly with growth in tests while when doing it manually the cost increases much faster as tests increase.
The benefits of automated testing is something that can make or break a company’s decision to begin utilizing automated testing. Luckily, in most cases, automation is much more efficient and cost-effective. While it may be a bit pricey to get started, it’s something that can be used again and again, with very little to no editing. And that time that was formerly spent on manual testing can be used for other, more useful things. Brilliant minds within the company can be put to good use, and the redirection of manpower can be a fantastic way for companies to save money. While it might seem that automated testing is complicated and expensive, its easy to see that it often has a glaringly positive return on investment, and does far more good than bad.
With Bqurious Platform your Test Automation RoI is increased manifolds due to several accelerators and support for several enterprise apps.
You can also look at this article for another view at calculating Test Automation RoI.
Reach out to us for a downloadable version of this RoI spreadsheet.