How to get most out of your software testing budget?

Published on March 9, 2018 - by sundars under testing

Software testing is becoming expensive particularly with resources cost increasing. Effective use of software testing budget is one of the challenges. Getting more done with fewer resources is ubiquitous, and here are some guidelines to get the most out of your software testing budget.

Following are the likely options you will have for your software testing budget:

  1. Hire in-house test engineer(s)
  2. Engage outsourcing vendor(s)
  3. Sign up for on-demand crowdsourced testing service(s)

Apart from the above options, you may additionally need to buy testing tools and devices. But we will keep them aside from this discussion because there are a lot of choices and options for tools.

Now, let’s focus on how to get the most out of your budget using the above options.

Setting your expectations and identifying the key metrics to track will be your first step to get the most value out of your budget.

Setting Expectations and Identifying Testing Metrics:

  1. Preventing bugs creeping into production should be your first expectation. Anybody you use for testing, must help you find more bugs sooner in your releases cycle
    1. Defect Removal Efficiency: The defect removal efficiency (DRE) gives a measure of the testing team ability to remove defects prior to release. It is calculated as a ratio of defects resolved to total number of defects found.
  2. Executing several cycles of testing in short span of time and increasing test coverage in terms of testing on more environment, browsers, devices and test data combination is critical. Also, time taken to execute each testing cycle should come down and stay on a plateau for the same number of test cases.
    1. Test cases written per hour.
    2. Test cases executed per hour.
    3. Estimated vs Actual Testing hours.
    4. Unplanned work vs planned work
  3. Balancing test executions and identifying new issues sooner in the release cycle while staying within budget is very critical. So, track your budget on monthly basis against the total work done.
    1. Defect Slippage Ratio: Number of defects slipped (reported from production) v/s number of defects reported during execution.
    2. Valuable work done vs budget burn down.

Pros and Cons of different engagement models:

Understanding pros and cons of your options will help you to make the informed decision. Let me try to present the pros and cons of the different engagement model that you may choose based on the aspects of productivity, efficiency, reliability and risk reduction.

Hire in-house QA engineerEngage Outsourcing VendorOn-demand Crowdsourced Testing
productivitypros: In-house resources can understand requirements and create intelligent test cases faster.

cons: Asking them to execute regression test cases and report on them will lead to productivity loss and in-effective use of the budget.



pros: Outsourced vendor executing regression test cases and reporting on them will be little faster because for the same budget you get more resources.

cons: Using outsourced vendor to understand requirements and writing intelligent test cases can lead to much communication and management overhead with productivity loss for in-house resources who have to manage and communicate with the outsourced vendors.

pros: Executing regression test cases and reporting will be so fast due to manual and distributed execution across the world.

cons: Quality of regression test execution results need to be validated which adds additional overhead for in-house resources. Moreover, crowd sourced resources can’t be used to write any test cases unless your application is simple and straight forward.

efficiency pros: If regression test execution part is taken care by an outsourcing vendor, in-house resources can spend more time in exploratory testing which may result in identifying more bugs.

cons: If the in-house resources are used for regression test executions and exploratory testing, they won’t be efficient in finding new bugs because they have to constantly switch between tasks. Find bugs will take time and it’s important to provide the undistracted time to your in-house engineers.

pros: Outsourced vendors can find bugs if they perform exploratory testing in addition to regression testing and balancing their work efficiently.

Cons: Outsourced resources should be managed by the in-house resource to great extent to achieve efficiency. Also, if outsourced resources need more handholding it will lead to loss of efficiency of in-house resources. Overall there is huge management overhead.

pros: If your application is straightforward and simple to understand and use, there are more chances for on-demand crowdsourced testers to find more bugs as more eyes are involved in testing.

cons: cost will be a huge barrier, as more resources need to be deployed you can quickly burn out your budget. Moreover, the in-house team needs to validate the bugs identified by crowdsourced testers.

risk reductionpros: if test execution part is taken care by outsourced vendor, the in-house resources can spend more time in exploratory testing which may result in identifying more bugs. Ever bug identified and fixed before release reduces risk.

cons: If the in-house resources are used for regression test execution, they won’t be efficient in finding new bugs which will increase the risk.

pros: Outsourced vendors can find more bugs if they are involved in the project long enough and have spare time to do exploratory testing which may result in identifying more bugs.

cons: When Outsourced vendors are completely used for regression testing, they may not add much value finding issues with new functionalities that are being built. This will increase the risk.

Same as above
reliability pros: Highly reliable because in-house resources can work closely with you.

cons: Reliability come at higher cost because.

pros: Outsourced vendors sometimes may not be reliable due to different work timing and human dependency, but still you can rely to some extent if its outsourced testing resources.

cons: Reliability come at higher cost and management overhead. If its new vendor, it takes time to establish the team and feel confident.

pros: good reliability in test executions due to scale through humans

cons: test execution reliability come at higher cost and in-house management overhead

Final thoughts:

Overall splitting your budget into two and engaging different players will help you to get the most out of your budget.

  1. For testing and assuring existing system functionalities (regression), use outsourced vendors or on-demand crowdsourced testers.
  2. For exploratory or ad hoc testing that is focused on finding new bugs, use in-house resources they are going to be effective.

Ideally, 40% budget for regression and 60% budget for exploratory or ad hoc testing will be an optimal split.

Sundar Sritharan

Linkedin Icon

Sundar Sritharan

Sundar heads the Development and Testing Engineering practice at Softcrylic with his 17 years of experience. He consults, architects and delivers digital transformation solutions using Agile and Lean development methodologies for clients across the globe.

Start typing and press Enter to search