Testing Strategies for Continuous Delivery

Published on May 11, 2016 - by sundars under testing

The shift towards Continuous Delivery in Software Development in many projects and organizations is reaping benefits such as faster feedback, vastly improved time to market, increased quality, and a better customer experience. The release of a new build at the push of a button has significant impact on the approach used to test these applications. The promised delivery speed of Continuous Delivery demands a testing pace that is challenging to keep up and support the speed of new releases.

Continuous Delivery

(CD) is a disciplined, integrated, and highly automated process to expedite the process of incorporating new code—from initial development to production release—with confidence that the new code will function as designed and improve the value of the product.

There has also been a move to Continuous Deployment. The primary difference between Continuous Delivery and Continuous Deployment is that the continuous delivery pipeline automatically tests the application, but keeps the deployment decision as a manual step. A continuous deployment pipeline, on the other hand, will automatically deploy the working version. There is no “right way” to deploy to production—it is a decision made by project teams.

continuous delivery continuous deployment - Softcrylic

Figure 1. Continuous Delivery and Continuous Deployment are essentially identical up to the last step.

In either case, when a developer commits a new bit of software to the version control repository, a suite of automated jobs compiles and packages, tests and deploys the code to development and test servers and stores the packaged code so it can be released with the click of a button. Should the code fail during any test phase along the pipeline, the process halts and sends the code back to the developers for fixes.

Business Benefits of Continuous Delivery.

Reduced Time to Market: CD significantly reduces—from months to weeks or days—the time it takes to develop, integrate, and validate new code into an existing software program.  CD increases the frequency of releases—from huge annual or semi-annual program releases to a weekly or even daily releases of specific new features, fixes, and upgrades.

The Right Product: Frequent, short-lead-time releases result in quicker customer feedback that allows marketing teams to quickly determine if a new feature is right for the market place. If not, unlike prior to CD, months of development and testing have not been wasted.

Improved Productivity: Because CD depends heavily on automated deployments, developers no longer have to spend time setting up and updating their test environments. Developers and operations engineers no longer have to spend time and effort troubleshooting. These steps are automated in the CD pipeline. And the pipeline automatically releases the new application to production when it is ready. This cuts weeks, even months, off of the develop-test-release cycle.

Improved Product Quality through Reliable Releases: Because of the emphasis on increased, automated testing, the CD pipeline inherently delivers continuous improvements to product quality through highly reliable releases. If the tests find a problem, the developers can fix it. The automated tests in the pipeline have resulted in as many as 90% fewer bugs released to production.

The CD pipeline also enables the ability to rollback to an earlier release version in the event that a serious problem is encountered—a bit of reassurance for both the customer and management when new software is released.

Improved Customer Satisfaction: All of these features provide customers with both a higher quality product and even more importantly, a higher level of confidence that the program will deliver the performance required.

The benefits of CD can be easily summarized: “It saves time.  It saves money.  Everyone wins.”

Role of Test Automation in Continuous Delivery

The CD pipeline is essentially a series of automated testing nodes that progressively test the build with unit tests, acceptance or functional tests, and performance tests prior to and after integration of new code in the production. At any point along the pipeline, if a problem is detected, the process stops and returns the code to the developers for a fix. An equivalent amount of manual testing would take months versus hours for CD.

Testing Strategies for Continuous Delivery

Figure 2. Test automation makes CD economically possible, from code commit to production release.

Code Commit: When a developer commits code to the source code repository, Continuous Delivery software such as Jenkins automatically compiles the source code and executes unit tests. When a unit test encounters an error, the pipeline stops, the Developers fix the code, and check in the revised code to restart the process. If everything goes well, the pipeline automatically moves the code to a Binary Repository and deploys the build to Acceptance Testing environment.

Acceptance Testing: The pipeline creates and executes a production-like acceptance test environment and deploys the application for automated acceptance testing. The pipeline then runs an acceptance test suite that ensures that the software meets all specified user requirements to validate the functionality of deployed application.

Performance Testing: The performance test ensures that the new code will not adversely affect the software’s performance. As before, the pipeline sets up the performance test environment, runs performance tests, and records the results in the software quality management tool. This process enables efficient performance testing on each piece of new code rather than the pre-CD practice of doing performance testing only just before a major release.

Manual Test:  Although the objective is to use test automation throughout the CD pipeline, manual testing is sometimes necessary, especially for exploratory testing and a business user’s acceptance testing.

Finally the application is released to production.

Recommended Test Strategies to Implement Continuous Delivery

The key to successful CD is a functioning pipeline of automated tests, but test automation is a distinct discipline in itself.

“Shift Left” Testing: “Shift left” testing refers to moving tests to the left—earlier in the schedule—on the traditional developmental time line. Trends in test driven development (TDD) and Behavior Driven Development (BDD) represent the ultimate in “shift left” testing because the developer writes the test scenario before actually writing code.  TDD and BDD require upfront design and thinking about the code required to achieve an outcome. Writing the test first influences the actual code written—it is more focused on the desired function, and more testable.  The idea is to write code guided by “how am I going to test this?”

Behavior-Driven Development is a development process that emerged from TDD. Behavior-driven development combines the general techniques and principles of TDD with ideas from domain-driven design and object-oriented analysis and design to provide software development and management teams with shared tools and a shared process to collaborate on software development. BDD is largely facilitated through the use of a simple domain-specific language (DSL) using natural language constructs (e.g., English-like sentences) that can express the behavior and the expected outcomes. BDD is considered an effective technical practice especially when the business problem to solve is complex.

Cross-Browser Testing: Assurance Tests are generally automated to target functional reliability. Time does not always permit manual cross browser testing for an application. Practical testing strategy suggests that testing be focused on the most popular and most used browsers and OS. Using application usage analytics data such as data from Google Analytics, we can identify which browsers’ end users mostly use. Determine a threshold of usage below which the value of testing is not warranted.

Using open source tools, such as Selenium WebDriver is an effective way to get started with test automation. Selenium WebDriver is a very popular functional automation tool and has evolved rapidly. WebDriver interacts with web applications in a very similar way as a human, providing a real-time simulation of user actions with the website. Selenium WebDriver is a portable testing framework for web applications that allows test to be written in many programming languages. The tests can then be run against most modern web browsers. It is open-source free software, released under the Apache 2.0 license. The Selenium WebDriver provides a framework that would support the CD pipeline.

Developers may also want to use a continuous build tool (e.g. Maven, Jenkins) where the latest test scripts are built and executed against the latest application build on a daily or periodic basis. An assurance of functional correctness is thus provided on an ongoing basis by running tests across browsers.

Performance Tests include both operational features (the features that are used to accomplish the user’s tasks) and Non-functional Performance Testing. Performance testing is quality assurance (QA) testing to ensure software applications will perform as well under their expected workload. A software application’s performance–like its response time–does matter. The goal of performance testing is not to find bugs but to eliminate performance bottlenecks.[1]

Non-functional features include features like system performance, security, reliability, usability, maintainability, and scalability, to mention only a few. Of these examples, however, only system performance testing and security testing are key to the performance of a program and should be considered priorities in the testing schedule and included in the Continuous Delivery pipeline.

The Apache JMeter™ application is open source software. It is 100% pure Java application designed to load-test functional behavior and measure performance. It was originally designed for testing Web Applications but has since expanded to other test functions. Apache JMeter may be used to test performance both on static and dynamic resources including: Webservices (SOAP/REST), Web dynamic languages (PHP, Java, ASP.NET, Files, etc.), Java Objects, Data Bases and Queries, FTP Servers and more). It can be used to simulate a heavy load on a server, group of servers, network or object to test its strength or to analyze overall performance under different load types. It can be used to make a graphical analysis of performance or to test server/script/object behavior under heavy concurrent load.

It is important to understand that different architectures and objectives may require different strategies for performance tests. For example, when deploying an API server that handles a lot of incoming requests from mobile devices or other applications, tests might focus mostly on throughput: the hits or requests per second that various endpoints can handle within given response-time expectations. Those tests can use straight URL requests without regard for the complexities of think time or extraneous logic that synthetically shape traffic.

Testing in Production: Unlike Acceptance and Performance Testing, testing in production is a way to judge user acceptance of a specific feature. When a new feature is launched, it may be released to only 10 percent of users of the software. Then the used in monitored for a period. This will be followed by an alternate version of the feature. User acceptance of the two versions are compared to determine which is used more, and eventually the feature is released to the to all users. This is often referred to as A/B testing or an Online Controlled Experiment.

A Partner in CD

Softcrylic, LLC has been at the forefront in providing Test Automation services for continuous delivery in an agile development environment. They have developed a hybrid test automation tool, Automate-On®, to aid organizations to get started quickly with test automation.

Errors or bottlenecks introduced by automated testing infrastructure can break builds and block your pipeline, creating expensive delays for software developers. Running automated tests rapidly and reliably is critically important to a successful Continuous Delivery process.

By providing a high-reliability, scalable automated testing platform, Softcrylic has been able to help enterprises to be up and running in test automation within days. Automate-On® has been specifically designed to fit into the ecosystem of popular testing frameworks, CI systems, and surrounding tools and services, so that users can leverage existing investments and focus on optimizing their CD pipeline.

Test Automateon e-Commerce Sites

Contact Softcrylic, LLC to for the best guidance on Test Automation and to realize the potential of Continuous Delivery in an agile environment.

References Cited

  1. http://www.guru99.com/performance-testing.html
  2. http://jmeter.apache.org/
  3. https://blogs.msdn.microsoft.com/seliot/2011/06/07/testing-in-production-tip-it-really-happensexamples-from-facebook-amazon-google-and-microsoft/
    Sundar Sritharan

    Linkedin Icon

    Sundar Sritharan

    Sundar heads the Development and Testing Engineering practice at Softcrylic with his 17 years of experience. He consults, architects and delivers digital transformation solutions using Agile and Lean development methodologies for clients across the globe.

    Start typing and press Enter to search