KPIs for software tests - quality assurance for your product

Developing software that solves a pain point for the user is undoubtedly the foundation of a successful software product. However, there is another aspect of utmost importance: software testing. Software testing is often neglected, even though the test result determines whether your product stands out from the competition - or not.

In this article, we take a closer look at the importance of software testing and how you can measure whether your product development is on the right track. We will introduce you to the most present key performance indicators (KPIs) for testing the software you develop, illustrate the theory of software testing with a practical example and provide you with important information on how you can gain a competitive advantage by testing your software.

Key performance indicators (KPIs) for software testing

There are many KPIs that reflect the success of the test processes. For the sake of brevity and implicit prioritization, we have selected the most important of these KPIs. Here is our selection:

KPI 1 "Requirements Test Coverage" - Requirements for the scope of testing

Requirements Test Coverage measures the degree to which the test cases executed during testing cover the set of specified functional and non-functional requirements for the Software cover. A prerequisite for the applicability of this key figure is that the requirements for the software have been fully specified.

Interpretation and application of test coverage as a KPI:

  • This KPI ensures that each requirement is linked to at least one test case that checks it. It is important to note that each requirement can be tested by multiple test cases. At the same time, a test can also check several software requirements. This means that there is a many to many relationship between test cases and requirements.
  • A 100 percent requirements test coverage does not mean that there are no other possible tests that can be written. It only means that each requirement is linked to at least one test case. However, there is no guarantee that this test case covers the entire specification of the requirement.
  • Don't forget: If the requirements for the software change, the test cases must also change.
  • To implement this KPI, you must Traceability tools to link test cases and requirements with each other. This traceability helps to track which requirements have already been tested and which are still outstanding. This traceability is used to measure coverage as a percentage.

Example "Requirements Test Coverage"

You are in a senior role at an e-commerce company that is planning a major upgrade of its e-commerce platform to improve the customer experience. This involved defining both functional and non-functional requirements. As part of the Quality assurance process it is crucial to ensure that all requirements are adequately tested.

Your development team uses traceability tools to link test cases and requirements. This allows you to measure test coverage and ensure that each requirement is covered by at least one test case.

After carrying out the tests and linking requirements and test cases, you find that the test coverage is 90%. This means that 90% of the specified requirements have been checked by at least one test case. This is a good result, but shows that there is still room for improvement.

When analyzing the test coverage, you notice that some of the non-functional requirements, such as the performance of the platform under high load, have not yet been sufficiently tested. In this case, you need to ensure that additional test cases are created to cover these specific requirements. Furthermore, it turns out that some of the test cases cover multiple requirements and vice versa. This illustrates the many to many relationship between test cases and requirements. If there are changes to the requirements during development, the test cases need to be adapted accordingly to ensure that the requirements test coverage remains up-to-date.

By using traceability tools and regularly checking the requirements test coverage, you ensure that the e-commerce platform meets the specified requirements and that potential problems are identified and rectified at an early stage. A requirements test coverage of 100% does not necessarily mean that no further tests are required, but that each requirement has been checked at least once.

KPI 2 "Code Coverage" - Coverage of the code

Code Coverage measures how much of your source code, including lines of code, instructions, branches and paths, was actually executed by your tests.

Interpretation and application of code coverage as a KPI:

  • Code coverage is usually expressed as a percentage, e.g. 20% or 90%. A higher percentage implies more extensive testing. There is no clear number that can be set as a target, but if the percentage is higher than 70%, this is a good value.
  • Code coverage tools often highlight the specific lines or branches of code that were not executed by your tests.
  • If you identify areas of undiscovered code, you can use this information to close test gaps. It is also possible to create additional test cases or modify existing ones to test these undiscovered code paths.

Example "Code Coverage"

Your e-commerce platform has recently undergone an update, adding new functionality and changing some existing features. As part of your quality assurance process, it is important to ensure that all code has been sufficiently tested, especially after such updates.

Your development team uses a code coverage tool to ensure that the test cases run through the code thoroughly. After the Update you perform a code coverage analysis and find that the code coverage is only 65%. This means that only 65% of the source code has been run through by your test cases.

The Code Coverage Tool highlights the specific lines and branches of code that were not executed by the tests. In this case, it is noticeable that some of the new functions and parts of the existing code have not been sufficiently tested.

To improve code coverage, take the following measures:

  1. Identification of test gapsAnalyze the highlighted lines and branches of code that were not reached by the tests and identify the areas with undiscovered code.
  2. Creation of new test casesYou create additional test cases that cover these undiscovered code paths. This may mean developing specific scenarios that thoroughly test the new features or modifying test cases to cover these new areas.
  3. Modification of existing test casesYou revise existing test cases to ensure that they cover the changed parts of the code.

These measures increase code coverage and ensure sufficient testing of the new functions and changed parts of the code.

KPI 3 "Test Automation Coverage" - Coverage of test automation

Test Automation Coverage shows you what proportion of all test cases available to you as part of your test set are automated. The metric is formed by simply dividing the number of automated tests by the total number of tests - automated and manual. It is important to note that all test cases to be considered are those that are executed at the system level - that is, in such a way that any end user of the software can execute them.

Interpretation and application of Test Automation Coverage as a KPI:

  • Test Automation Coverage is expressed as a percentage. A higher percentage means better coverage and vice versa.
  • Test automation is not a quick process, it takes time and therefore requires good planning. To know which tests should be automated first, you need to set priorities. For example, consider the frequency of use of the tested functionality, the dependency on other test cases and the complexity of the implementation of your software. Test cases with high priority, strong dependencies and low complexity are good candidates for automation.
  • Test automation is never-ending as long as you continue to develop your product. Don't forget to regularly review and update your automated test cases to reflect the latest changes in functionality and performance.

Example "Test Automation Coverage"

Let's stay with our company, which operates an e-commerce platform where customers can buy products. Their platform has a variety of functions, including product search, shopping cart, order processing and customer reviews. As Quality Assurance Manager, you are responsible for overseeing test automation and ensuring good test coverage.

Your e-commerce platform has a total of 500 different test cases that are relevant to the system level. These are tests that ensure that an end user can use all the important functions of the platform. To calculate your test automation coverage, you have automated 200 of these tests. This means that 200 of your test cases are executed by a test automation framework, while the remaining 300 are executed manually by testers.

The Test Automation Coverage, in this case, is:

(automated test cases / total number of relevant test cases) x 100 = (200 / 500) x 100 = 40%

This means that your Test Automation Coverage is 40%. This in turn means that 40% of your relevant test cases Automated while the remaining 60% are performed manually. This may be due to various factors, such as time and resource constraints or the difficulty of automating certain tests.

The following steps can be taken to improve Test Automation Coverage:

  1. PrioritizeAnalyze which tests are performed most frequently by users, such as product search and order processing. These tests have the highest priority for automation.
  2. Take dependencies into accountIdentify test cases that are highly dependent on others. The automated tests should be those that have the greatest impact on other tests to increase efficiency.
  3. Consider complexityTests that are easy to automate should be preferred as they require less time and resources.
  4. Regular updatingYou ensure that the automated test cases are constantly revised and adapted to the latest changes in the platform in order to keep the test coverage up to date.

By applying these principles, you will be able to increase test automation coverage and ensure the quality of your e-commerce platform while using time and resources efficiently.

KPI 4 "Defect Detection Efficiency - DDE" - Efficiency of defect detection

Defect Detection Efficiency (DDE) is expressed as a percentage and represents the proportion of defects or bugs detected through testing compared to the total number of defects/bugs in the Software represent.

The formula for calculating the defect detection rate is as follows:

Error detection rate (%) = (number of errors found during testing / total number of errors) x 100

Interpretation and application of the DDE formula as a KPI:

  • A higher percentage means that the tests are more effective. It indicates that your test procedure is sufficient and that only a minimal number of defects will reach the production environment.
  • A low percentage means that your testing process is less effective at finding bugs and you need to expand your test set. When deciding how to do this, you can use the data provided by your code coverage tools.

Example "Defect Detection Efficiency"

A software development team is working on a new e-commerce system. During the development phase, 100 errors or bugs were identified in the code. Before the software goes into the production environment, extensive tests are carried out to identify and fix the errors. During these tests, 80 of the 100 errors are found and fixed.

To calculate the DDE, we use the formula:

DDE (%) = (number of errors found during testing / total number of errors) x 100

Let's insert the values into the formula:

DDE (%) = (80 / 100) x 100 = 80%

In this example, the Defect Detection Efficiency (DDE) is 80%. This means that 80% of the existing defects were found during the tests. A high DDE indicates that the testing efforts are efficient. Only 20% of the defects could enter the production environment undetected.

In practice, the development team would strive to keep the DDE as high as possible to ensure software quality and minimize customer complaints or quality issues after release. If the DDE were lower, the testers and developers would rethink their testing strategy and possibly increase test coverage to ensure better defect detection. The use of code coverage tools, among other things, could support this and identify the areas in the code that have not been sufficiently tested.

KPI 5 "Escaped Defects

The term "escaped defects" refers to errors or problems in a Software applicationthat were not detected and rectified during the test phase and subsequently entered the production environment. They "escaped" the test process and were discovered by the end user. End users are not only customers, but also internal participants who came across the error outside the test process.

Interpretation and application of escaped defects as a KPI:

  • Escaped defects can lead to higher maintenance costs. Fixing defects in the production environment is often more costly and time-consuming than fixing them in the test phase.
  • These errors can have a negative impact on the user experience.
  • The presence of escaped defects may indicate deficiencies in the testing process, such as insufficient code coverage, insufficient coverage of requirement tests, or a lack of effective test cases.

Example "Escaped defects"

Your e-commerce company has recently launched a new version of your e-commerce platform into the production environment, which included some significant changes and improvements. Prior to release, the software went through an extensive testing phase where the developers and QA teams carefully reviewed the requirements, code coverage and various test cases.

A few weeks after the release of the new version, you start to receive reports of problems and bugs from end users. These bugs affect the user experience and lead to customer complaints. Upon investigating these missed bugs, you discover that they are due to unrecognized vulnerabilities in the production environment. These vulnerabilities were not detected during the testing phase and have made it into the live environment.

Such a missed error is, for example, an error in the ordering process that results in orders not being able to be completed correctly. This significantly impairs the user experience and leads to lost sales, as customers are unable to complete their purchases successfully.

The consequences of missed errors are manifold:

  1. Higher maintenance costsFixing bugs in the production environment requires additional resources and time. This leads to higher maintenance costs as developers and support teams have to work urgently on bug fixing instead of working on new features.
  2. User experienceMissed errors affect the user experience and can lead to customers being dissatisfied and possibly switching to competitors.
  3. Quality assurance processMissing defects indicate possible deficiencies in the testing process. This may indicate insufficient code coverage, insufficient coverage of requirements or ineffective test cases. It is important to review and improve the testing process to avoid such errors in the future.

As a result, it is important to analyze the missed bugs, fix them and at the same time strengthen the testing process so that such issues are avoided in future software updates. This will help reduce maintenance costs and maintain user confidence in your e-commerce platform.

KPI 6 "Customer Satisfaction" - Customer satisfaction

Even if customer satisfaction is not directly and exclusively related to software testing, you should consider this KPI both when developing and Optimization of a software testing strategy. Customer Satisfaciton is a measure of how well your software meets user expectations. In this way, this KPI can show you - in addition to the findings about the functionality itself - how high the perceived quality of your software is.

Interpretation and application of customer satisfaction as a KPI:

  • There are many surveys that customers are asked to complete on a regular basis that provide a numerical value for customer satisfaction. Examples: Net Promoter Score or Customer Effort Score.
  • Although numerical values are easy and quick to understand, you should not rely on these alone. Instead, make sure that you also integrate qualitative questions. These answers will show you the "why" behind the number, which in many cases is not a missing function, but simply an unreliable software product full of anomalies.
  • Evaluate the relationship between the problems identified and your testing procedures. Assess whether these issues could have been detected through more comprehensive or targeted testing. This could include functional testing, usability testing, performance testing or security testing.

Example "Customer Satisfaction"

Your e-commerce company conducted a customer survey and used the Net Promoter Score (NPS) to measure customer satisfaction. The NPS resulted in an average score of 7. Although this is positive at first glance, qualitative responses showed that customers had issues with website performance, which affected their shopping experience.

The qualitative analysis showed that these performance problems affected the perceived quality of the software. The cause was insufficient performance testing during development. In response, they optimized the test strategy and integrated more comprehensive performance tests.

The Customer Satisfaction KPI helped to identify the gap between customer requirements and testing procedures and adapt the testing strategy to increase customer satisfaction and ensure the long-term success of the company.

Conclusion on KPIs for software tests

Software Tests KPIs Conclusion

Software testing often proves to be a neglected aspect of software development. Yet testing your software plays a central role in the success of a product. There are a variety of KPIs to assess how well the tests are performed for each iteration.

It is unrealistic and not even necessary to apply all KPIs. However, they often complement each other and should be considered in combination.

For example, it is possible to track the development of the performance indicator for Escaped Effects across the different versions of the software. This KPI ultimately determines the success of all your test strategies. However, other KPIs such as Code Coverage, Test Automation Percentage and DDE will help you decide which parts of the software you should focus on in your efforts to improve your testing strategy if the number of escaped effects increases.

Note!Please also consider other software development best practices when implementing these KPIs, such as investing time and effort in requirements analysis to obtain a complete set of requirements.

Do you have any questions about software testing? Write us a message.

    Boyidar Parvanov Avatar

    Latest articles