Data has increasingly become a crucial part of the way our world functions today. In the highly connected digital world, data is key when it comes to the applications being used by consumers across the globe. A single web page can have a multitude of data entries with thousands of inputs and accounting for this aspect in the development phase needs to be undertaken to ensure a well-functioning webpage or any application that has multiple input possibilities.
Testing against a large number of input points hence becomes an absolute necessity to bring about robustness. Data driven testing is a way for software developers to speed up the process of validating that their code works as expected.
There are instances when the testers and developers need to conduct the same tests for a number of different data sets, for example, as noted above, a web page that has multiple entries and input values for a large pool of users. Creating a single test for each dataset is an impractical approach that would lead to excessive wastage of time and resources. It is in situations like these, in which there is a need for conducting iterative tests, that Data-Driven Testing comes to the rescue of developers and testing teams everywhere.
So, what exactly is Data-Driven Testing or DDT? It is a test design and execution methodology used in software testing in where data is read from data files rather than hard-coded values. Here, data is decoupled from the test. In the data-driven testing approach, the test environment control and settings are both not hard-coded. Data-driven testing is named so because of the fact that data is what these tests rely on.
Under DDT, the test in question is parameterized and run for a variety of input values, hence it is also known as parameterized testing.
Let’s take, for instance, the case of a developer who needs to test the functioning of an application. He inputs a certain set of information points with a successful result, and when using another set of data points, the test returns a failure. Now imagine he needs to do this repeatedly for an extremely large number of such permutations. It would be too cumbersome to keep doing this and to alongside, also maintain the records of the output.
If the test could itself pull out different permutations from a pool of data points, including various test scenarios, would that not save the developer an enormous amount of effort and time; effort and time that would ultimately lead to better usage of resources of the organization? That is what Data-Driven testing is all about.
It is essentially a four-step process as follows:
- Documenting the input values as well as expected outcomes in a file or a database.
- Creating test that can read the data from the data source and running the test on a set of input values.
- Storing the retrieved result and comparing it with the expected output as per the stored external data.
- Repeating the test and storing of test result for the next set of data points.
Any testing methodology that is not data-driven will include a sample dataset within the test, whereas, a data-driven test will connect to an external source for extraction of data for testing purposes.
Data files may include sources in a variety of formats like ADO objects, ODBC objects excel spreadsheets, CSV (comma-separated value) files, XML, as well as fully-featured databases like MySQL. In the DDT, the input data file will contain all possible values including the positive and negative test scenarios as well as the best and worst-case test scenarios. Additionally, for each varied data set being used as input for the test, the expected outcome is also included. The testing framework then assesses the test result and based on whether the actual result matches with the expected one, it passes or fails.
As noted above, positive and negative test case scenarios are both built into the data-driven testing framework. The idea of positive and negative test cases can be understood with the help of an example: say we have a webpage that requires a user to enter her date of birth. A positive test scenario is when a date like 27/05/92 as entered by the user would be accepted by the system and we can then say that our positive test case passed.
In the negative test scenario, if we have a date of the form 30/31/32 and the test returns an error, as it correctly should, the system passes the test. In the case that it accepts an invalid date of birth (30/31/32) as valid input data, we would say that our negative test has failed.
Imagine how much effort is saved when we use an external data source to test for positive and negative cases and how much it becomes easier to handle errors in our tests.
Multi-fold benefits of employing the Data Driven Testing methodology:
- The main advantage in undertaking data driven testing is the ease with which new inputs can be added or existing inputs can be modified to the product or system that is being tested, without the need to recreate the entire test to adjust for these changes since DDT makes use of data that is external to the functional test at hand.
- As a corollary of the above point, any changes made to the test logic or script will not impact the test data, which is, again, a major advantage of DDT.
- Since the input data is drawn from an external source and is not written into the code of the test itself, the script can be generated with less code making it easier to maintain.
- There is enhanced coverage since a single test is created and that very test is conducted with as much varying input data as desired. Increased test coverage is always a game-changer in the software development cycle.
- It allows for effective separation of test from the test data (decoupling), since the latter is from an external source. The data: inputs as well as outputs can be stored together in a manageable form. This separation allows for treating test and data as distinct units that can be used, altered and re-used independently.
- DDT reduces redundancy and improves efficiency in conducting tests that are repetitive or iterative in nature. It would make sense to use the data driven approach to testing when you need to run the same tests for a large number of varied set of inputs.
- With data driven testing, since multiple tests do not need to be created for each different set of data, the development teams are able to better allocate their time, effort and resources. For example, test team can create the test while the business users can provide the test data.
- The possibility of building positive and negative scenarios into the input data itself proves to be advantageous since it takes into account all possibilities and makes the testing process more efficient.
- Apart from the positive and negative test cases, the expected test results are also part of the input data pool. This enables effective output validation, i.e., the comparison of actual results with expected ones.
- Another advantage comes from the fact that the scripting process of tests can be undertaken during the development phase of an application. Data collection and consolidation, and creation of the data source files can be done even before the testing begins, leading to improved utilization of the testing team’s time and resources.
Despite the myriad gains that result from data-driven testing, it still does pose a few challenges for the testing teams. Let’s take a closer look at the limitations of the DDT approach:
• The skills required for undertaking data-driven testing are usually high-level technical skills. Generally speaking, greater expertise and experience of the scripting language is needed and the tester may, at times, need to learn a new programming language altogether.
• For data-driven testing, the expected outcomes for each data set need to be defined. In the instances where this is too complex a task to undertake, the benefits of data-driven testing might be outweighed by the amount of effort required.
• In the DDT approach, there is a need to maintain large databases of all test data to be utilized, which can be a considerable effort.
• For test scenarios that have multiple input values that may be entered by the end-user, there will be a sizable number of data files for each test case.
• There is increased documentation needed since Data-Driven Testing uses the modular way to testing. Documentation surrounding script management, test results obtained, etc. needs to be stored with the aim of ensuring effective communication of the workflow amongst team members.
Even with the limitations noted above, Data-Driven Testing is an important approach to testing since it ensures greater test coverage, reduces redundancies in the testing process, and generally improves efficiency. Especially in cases where the application in question is such that it needs to be tested against a large number of data regularly, Data-Driven Testing becomes an obvious choice.
Bqurious tries to address the above limitations with its easy-to-use code-less Test Automation Platform. It offers the ability to create Data-Driven Tests without writing a single line of code and also the ability to seamlessly integrate with multiple external data sources like XLS, databases, etc. Bqurious also allows you to fetch data from REST APIs to be used in Functional Tests.