Software testing is becoming expensive particularly with resources cost increasing. Effective use of software testing budget is one of the challenges. Getting more done with fewer resources is ubiquitous, and here are some guidelines to get the most out of your software testing budget.
Following are the likely options you will have for your software testing budget:
Apart from the above options, you may additionally need to buy testing tools and devices. But we will keep them aside from this discussion because there are a lot of choices and options for tools.
Now, let’s focus on how to get the most out of your budget using the above options.
Setting your expectations and identifying the key metrics to track will be your first step to get the most value out of your budget.
Understanding pros and cons of your options will help you to make the informed decision. Let me try to present the pros and cons of the different engagement model that you may choose based on the aspects of productivity, efficiency, reliability and risk reduction.
|Hire in-house QA engineer||Engage Outsourcing Vendor||On-demand Crowdsourced Testing|
|productivity||pros: In-house resources can understand requirements and create intelligent test cases faster.|
cons: Asking them to execute regression test cases and report on them will lead to productivity loss and in-effective use of the budget.
|pros: Outsourced vendor executing regression test cases and reporting on them will be little faster because for the same budget you get more resources.|
cons: Using outsourced vendor to understand requirements and writing intelligent test cases can lead to much communication and management overhead with productivity loss for in-house resources who have to manage and communicate with the outsourced vendors.
|pros: Executing regression test cases and reporting will be so fast due to manual and distributed execution across the world.|
cons: Quality of regression test execution results need to be validated which adds additional overhead for in-house resources. Moreover, crowd sourced resources can’t be used to write any test cases unless your application is simple and straight forward.
|efficiency||pros: If regression test execution part is taken care by an outsourcing vendor, in-house resources can spend more time in exploratory testing which may result in identifying more bugs.|
cons: If the in-house resources are used for regression test executions and exploratory testing, they won’t be efficient in finding new bugs because they have to constantly switch between tasks. Find bugs will take time and it’s important to provide the undistracted time to your in-house engineers.
|pros: Outsourced vendors can find bugs if they perform exploratory testing in addition to regression testing and balancing their work efficiently.|
Cons: Outsourced resources should be managed by the in-house resource to great extent to achieve efficiency. Also, if outsourced resources need more handholding it will lead to loss of efficiency of in-house resources. Overall there is huge management overhead.
|pros: If your application is straightforward and simple to understand and use, there are more chances for on-demand crowdsourced testers to find more bugs as more eyes are involved in testing.|
cons: cost will be a huge barrier, as more resources need to be deployed you can quickly burn out your budget. Moreover, the in-house team needs to validate the bugs identified by crowdsourced testers.
|risk reduction||pros: if test execution part is taken care by outsourced vendor, the in-house resources can spend more time in exploratory testing which may result in identifying more bugs. Ever bug identified and fixed before release reduces risk.|
cons: If the in-house resources are used for regression test execution, they won’t be efficient in finding new bugs which will increase the risk.
|pros: Outsourced vendors can find more bugs if they are involved in the project long enough and have spare time to do exploratory testing which may result in identifying more bugs.|
cons: When Outsourced vendors are completely used for regression testing, they may not add much value finding issues with new functionalities that are being built. This will increase the risk.
|Same as above|
|reliability||pros: Highly reliable because in-house resources can work closely with you.|
cons: Reliability come at higher cost because.
|pros: Outsourced vendors sometimes may not be reliable due to different work timing and human dependency, but still you can rely to some extent if its outsourced testing resources.|
cons: Reliability come at higher cost and management overhead. If its new vendor, it takes time to establish the team and feel confident.
|pros: good reliability in test executions due to scale through humans|
cons: test execution reliability come at higher cost and in-house management overhead
Overall splitting your budget into two and engaging different players will help you to get the most out of your budget.
Ideally, 40% budget for regression and 60% budget for exploratory or ad hoc testing will be an optimal split.