Leading an Organization into Continuous Delivery can be a big task. Nor canbusinesses just “turn a switch” and start doing Continuous Delivery (CD)overnight. This guide attempts to cover the main points that organizationsneed to consider when starting to implement and adopt Continuous Delivery:

• Continuous DeliveryPipeline * Continuous Integration and ContinuousDelivery * Continuous DeliveryTools * Continuous Delivery BestPractices * Continuous DeliveryTesting * Continuous Delivery TestStrategy * How to Get Started with ContinuousDelivery The fundamental underpinnings of CD are dependent on two parts: culture changeand commitments to the tool chain. A larger part involves commitment to thetool chain. This is because Continuous Delivery requires an increase inAutomation-both Task Automation and Test Automation. Each section containseverything businesses need to consider when it comes to building andimplementing a successful continuous testing program; from tool chain, to bestpractices, to developing a strategy, and discussing options on how to getstarted.

Continuous Delivery Pipeline

Software development has become fully mainstreamed in the last few years,largely due in part to DevOps. Every modern business organization has asoftware development team, just as it has an accounting team or an HR team.These teams are all predictable and deliver a known product. With DevOps,there has been an increased pressure on development teams to work this way.The idea is that with software development being mainstreamed, it needs to bemore predictable, more manageable and more responsive to the business.Software Development is now expected to deliver business value and can nolonger be a “mysterious black box.”

What is the Continuous Delivery Pipeline?

First, let’s establish the fact that Continuous Delivery combines some XP(eXtreme Programming) and Lean Manufacturing Principles. They include:

• Small Releases * Continuous Integration * Collective Ownership * Deliver as fast as possible * Quality at every step * See the whole Many people have started to look at software development in terms of amanufacturing production line. While there is great debate about this topicfor test teams, in many organizations—like it or not—it has arrived. The lean,XP, and manufacturing production line ideas (coupled with tools being usedtoday) are pushing us toward product development as a factory production line.Tool chains can automatically flow chunks of code from environment toenvironment, getting more production-like as it flows. This leads to smallreleases that are tested at each step that are frequently and quicklydelivered for deployment. The Infrastructure as code movement (the idea of demystifying infrastructureand turning it into a reproducible and consistent practice by task Automation)has already arrived and the use of cloud services is making its movementfaster. An important note here is that for many teams, re-engineering theproduct architecture to fit these paradigms is very often a large anddifficult problem in itself. The tools, infrastructures, practices, and newskills associated with Continuous Delivery are pushing product development tobe more stable, and repeatable—a consistent production line that producesimmediate feedback. The pipeline flows from build to test, then it getspromoted to new environments with testing at every step. Ultimately, the newchunks on the production line are deployed to production. All this happenswith tool-enabled infrastructure as code. Continuous Delivery has been related to a production pipeline. The main ideais that the “Continuous Delivery pipeline” is the image and description ofproduct flow through the development process and tool chain. This pipelineflow is a variation of a manufacturing production line. It became very clearin Agile that quality is everyone’s job. Likewise, everyone owns a piece ofquality. In CD, everyone owns deployment and everyone shares a piece ofdeployment. Many things must change about the product to get these ideasimplemented: architectural changes, tool changes, learning new skills tofacilitate, culture changes, rethinking the idea release to smooth out, andease deployment. As with all production lines, there will be bottlenecks. Thegoal with pipelines is a constant flow of high quality andfinished/deployment-ready work. Keep the flow going, Keep the pipeline moving.Get lots of and immediate feedback to the team. Does it work? Are customersusing it? What are the actual use cases? As Jez Humble from ContinuousDevelopment says, the solution is to adopt “a more holistic, end-to-endapproach to delivering software.”

Find Inefficiencies. Manage Constraints. Eliminate Bottlenecks.

Part of the production line and pipeline mentality is looking for bottlenecksand inefficiencies. This is managing by constraints. Whatever is slowing downthe deployment has to be addressed and fixed immediately to keep the flowmoving. As a warning for test teams, if the flow is constrained by testingand/or due to a lack of Test Automation, the team has to fix it. We willdiscuss this later in the Continuous Delivery Testing and Continuous TestStrategy sections.

Continuous Integration and Continuous Delivery

First, let’s establish the define Continuous Delivery and ContinuousIntegration and establish the differences. According to Wikipedia, ContinuousIntegration (CI) is defined as “A software engineering practice in which thechanges made by developers to working copies of code are added to the mainlinecode base on a frequent basis, and [is] immediately tested.” The goal is toprovide rapid feedback so that if a defect is introduced into the mainline, itcan be identified quickly and corrected as soon as possible. ContinuousIntegration software tools are often used to automate testing and build adocument trail. Because CI detects deficiencies early on in development,defects are typically smaller, less complex, and easier to resolve. In theend, well-implemented CI reduces the cost of software development and reducesthe time to market. Contrasted, ContinuousDelivery is a “SoftwareEngineering approach in which teams produce software in short cycles, ensuringthat the software can be reliably released at any time. It aims at building,testing, and releasing software faster and more frequently. The approach helpsreduce the cost, time, and risk of delivering changes by allowing for moreincremental updates to applications in production. A straightforward andrepeatable deployment process is important for Continuous Delivery”(Wikipedia). The main difference is that CI focuses on Dev and Test teams. CD focuses onOps teams. Where CI ends, CD picks up; promoting builds, versions, anddeployments up closer to and finally, the live/production environments.

Continuous Delivery Tools

The financial commitments to the tool chain often require a large upfrontinvestment for organizations, this can be hard due to the necessity ofincreasing task Automation. Tools will be needed for the following areas:

• Cloud/EaaS, Iaas, Paas * Configuration Management (e.g. Chef and Puppet) * Containers (e.g. Docker and Packer) * CI/Version Control (e.g. Jenkins, Bamboo, TeamCity, and Buildbot) * Deployment (e.g. XebiaLabs Deploy, Go Thoughtworks, IBM UrbanCode Deploy,and Code Stream) * Monitoring (e.g. New Relic andApp Dynamics) * Soure Control (e.g. Git SVN) * Test Automation (e.g. Selenium, TestArchitect, Cucumber, and jBehave)

Continuous Delivery Best Practices

The three best practices of Continuous Delivery are shift-left, Lean SoftwareDevelopment’s principles of Quality at Every Step, and Eliminating Waste andAutomate –Automate- Automate.

Shift-left

Shift-left is the most important best practice to discuss. For thoseunfamiliar, Shift-left is an approach to Software Testing and system testingin which testing is performed earlier in the lifecycle—this is a departurefrom Agile and is part of the maxim: “Test early and often” It is important to note that the idea of shift-left has been around for a longtime. It is grabbing hold much more today. In CD, organizations need to thinkabout shifting all testing, (security, performance, functional, and etc.) asearly as possible in the Dev Cycle, no longer to be done as an afterthought atthe end. The biggest impact of shift-left in CD is getting developers to initiatetesting earlier with more TDD, unit testing, code review, and other developertests. Doing this shifts some functional testing as early as possible. It isjust as beneficial to shift UI tests earlier to the API level whereverpossible. The idea of shift-left leads us to our next best practice…

Lean –Quality at Every Step

According to Wikipedia, Lean software development(LSD) is “atranslation of lean manufacturing principles and practices to the softwaredevelopment domain. Adapted from the Toyota production model, it is emergingwith the support of a pro-lean subculture within the Agile community. TheSeven lean principles are defined as the following:

1. Eliminate waste 2. Amplify learning 3. Decide as Late as possible 4. Deliver as fast as possible 5. Empower the team 6. Build integrity in 7. See the whole. Steps 1, 4, 6, and 7 are the most widely adopted in CD. The principle of“Eliminate Waste” is probably the biggest factor for success in a ContinuousTesting implementation. The idea of Testing in Continuous Delivery is thatAutomation regression suites are supposed to be “lean and mean” not bloated.The goal of Continuous Delivery and DevOps is to provide immediatefeedback-aka principle 4—deliver as fast as possible. Teams will need toredesign, optimize, and even cut the size of their automated suites forTesting in Continuous Delivery. While there is no set standard on “idealtimes” there is one guiding principle—if your regression suite takes days,maybe weeks to run, then it is too slow for CD. Building integrity-in issometimes referred to as “Quality at Every Step.” This means exactly what itsounds like. You have to build quality-in at every Step. This means qualityuser stories, quality code, quality unit tests, quality test cases, qualitybugs, quality automated scripts, quality performance tests, qualityenvironments, quality data, and all other deliverables at every step.

Automate Everything vs. Automate Smartly

In Continuous Delivery, everything needs to be Automated or better. Everythingthat can be automated needs to be. This falls into two categories; both taskautomation and Test Automation. Task Automation largely deals with deploymentand pipeline Automation, whereas Test Automation has several focuses inContinuous Delivery, most notably: performance testing, functional testautomation, and security test automation. With this, comes pressure on testteams. According to the Testing in Continuous DeliverySurveywhen asked “Do you feel that there is more pressure to automate recently?” 60%of respondents stated that “yes, a lot more.” Contrasted, 4% of the groupwrote “All of our tests are automated. There is no pressure.” It is prudent that teams don’t fall into the “automate everything” mindsetwhen it comes to Test Automation. Focus instead on smarter Test Automation, asopposed to more and more, and communicate this idea.

Continuous Delivery Testing / Testing in Continuous Delivery

Without a successful Automation practice in place, any team that attemptsDevOps is headed for disaster. DevOps will highlight the shortcomings of ateam on a larger scale, faster than Agile ever could. The reality today is that in Silicon Valley, and other places, many companieshave done away with QA departments in total. The Testing is done byprogrammers at the unit level and with the use of Microservices andcontainers. Testing is done with APIs at the API/services level and there islittle to no Testing done at the UI level. This will impact product quality,but to a much lesser degree than it has in the past where unit testing wasmysterious, if done at all. In these organizations, unit testing is a formalrepeatable, meaningful practice, and a gate for any step in the deploymentpipeline. The figure below illustrates all the places where Test Automation appears inthe Continuous Delivery Pipeline. In Continuous Delivery, Testing appears at every stage of the pipeline, it isno longer isolated. It is important to note that test teams, including TestAutomation, cannot become the bottleneck in the Continuous Delivery pipeline.The main reason is that if it does, it will be “fixed” and you may not haveany input on how it is fixed. For the product pipeline to keep moving with immediate feedback does not meantaking one Automated regression suite and having it run again, and again, andagain. That doesn’t ensure quality or confidence. There are certain times,particularly with newly committed code, that unit tests are most efficient forimmediate feedback. There are other times where API or service level testingare the best strategies and still other places where the tests best insertedinto the pipeline are UI driven tests.

Continuous Delivery Test Strategy

What is commonly referred to as a “Continuous Delivery Test Strategy” is inactuality Continuous Testing. Continuous Testing (CT) is the process ofexecuting Automated tests as part of the software delivery pipeline to obtainimmediate feedback on the business risks associated with a software release.(Wikipedia) and has itsown Automation strategy. It is important to remember that as you build testsuites for the CD pipeline, Test Automation does not show a lack of bugs. TestAutomation shows consistency with the last iteration; it shows stability andpredictability. Not finding bugs does not indicate that there are no bugs—itindicates that the bugs that haven’t been found yet are still hidden. What test to run and where to insert them—at what readiness? At whatinterface? In what environment? Against what data? When? These are allquestions that will become the foundation of your Continuous Delivery Testingstrategy. There are 3 elements that form your strategy:

• Environments * Regression Automation Suites * Testing in Production The test strategy for Continuous Testing is a full reexamination of what getstested where. This with the Lean Software development (LSD) idea of “qualityat every step” is the foundation of a CD test strategy. It involvesreexamining, what you have been calling, “regression” and developing multiplelean and mean suites for immediate feedback. It also means rethinking what and how you automate. Old style Automation—evensmall isolated automated tests need to be reexamined and replaced byefficient, low level tests as well as longer customer-focused transactionworkflows. Rethinking regression and Automation is a big task for even themost effective teams.

Environments

In Agile or Scrum, Dev or Test Teams “progressed” through environments, inContinuous Testing it’s important to consider what and why we test ondifferent environments: In DevOps/CD, you might be running your Automation on many or all theseenvironments concurrently. For example, what tests need to get run on thebuild environment? Why? What tests get run in the test environment? Why? WhatAutomated tests get run against, for example, VMs with various configurations,languages, browsers, mobile devices, etc.? What Automated tests get run on thestaging server? What is their purpose? What gets run on the staging server?What gets run in production? These are all important considerations to makewhen developing your testing strategy.

Regression Automation Suites

The term regression is probably the most misunderstood in all softwaredevelopment. The fact is, regression means many things to different people.Rethinking regression and Automation is a big task for even the most effectiveteams. The goal of Continuous Delivery and DevOps is to provide immediatefeedback (aka principle 4—deliver as fast as possible). To set the record straight, a regression test is any test you run again. Somepeople in development think the regression suite or full regression is runningevery test run against a previous build or product. That is rarely the case.Their regression suite is more likely rerunning every Automated test. For some teams, their automated “full regression” suite can take days to run.Many “full regression” suites have gotten bloated and slow, like we mentionedearlier in the best practices section, this is not ok. There is a speed toContinuous Delivery and Continuous Deployment that leaves no room for “back upplans”. The new description of regression has to be “lean and mean”, fast andfurious, bug finding! Not bloated or the same old “whatever” we automated atthe time. So for teams that are looking to implement CD, cleaning out theirtest suites to be lean and mean is a priority.

Testing in Production

It has always been a strict rule: “No testing in production”. The reason beingthat the last thing you wanted was to have the test team mess around on a livesystem or environment with customers only to find out that the Testing didsomething to muck up the system. This has entirely changed in Continuous Testing. In CD, you may be asked torun a suite of tests to validate the migration to live or monitor the livesystem. Whatever the case, you need to make sure you are testing what needs tobe tested and no more than you need to on live. Make sure the whole team knowsthe risks of testing in the live production environment, the risk of nottesting in the live environment, as well as that tests cannot interfere withthe normal user’s workflow performance or operation of the system-betransparent.

Getting started with Continuous Delivery

Continuous Delivery requires both high test coverage levels and an assurancethat their practices are in good shape. It is important that businesses do notunderestimate the difficulties of Automation. According to the testingessentialssurvey, released by LogiGear Magazine, almost half of respondents answered that thebiggest phase or aspect of testing that is not understood by management isthat “Test Automation is not easy”. Automation, and to a larger extent DevOps,can’t be successful if there isn’t serious time and investment from the start. According to the Test Automationsurveyreleased by LogiGear Magazine, only 19% of respondents who did a TestAutomation implementation were able to do it the successfully the first time.The Test Automation survey also revealed another shocking insight. When askedwhat would prevent them from Automating more, the top five results were:

1. “Not enough technical skill to build more successful Automation.” (34% ofrespondents) 2. “Investment in Automation program (training, staff, tool maintenance) istoo high.” (32% of respondents) 3. “Management does not understand what it takes to successfully automate.” (31% of respondents) 4. “Tool use cost is high.” (31% of respondents) 5. “Code or UI is too dynamic to automate more.” (31% of respondents) For argument’s sake, when looking at these problems, they largely deal witheither staffing or tool use. Commitment to the tool chain, as mentionedearlier is often expensive. While there are many tools on the market, thereare open source solutions like Selenium, or our ownTestArchitectthat can be used for Continuous Testing. TestArchitect integrates with mostleading tools including: Microsoft Team Foundation Server/ Visual Studio/ TestManager, Oracle EBS, HP Quality center, Jira, Jenkins, Zephyr, Neoload andmore! TestArchitect is a closed-source tool, but is available for free.TestArchitect can also handle rapid changes to the UI—it uses theAction-Based Testingmethod (ABT),which focuses on automating keywords, not test cases. Since there are fewerkeywords than there are test cases, keyword implementations tend to be shorterthan test case implementations; the Automation effort is also more manageable.In a well-designed keyword-based test suite, only a limited number of keywordshas to be maintained, after which then all tests run again against the newversion of the application. For staffing there are several options. If you have the requisite staffin-house but are lacking the technical skill for building a successfulAutomation program, it may make sense to consider engaging with a testingvendor. The biggest benefits of outsourcing testing are:

• More Automation * More effective testing * Faster product release * And more management oversight Testing vendors like LogiGear can provide the necessary expertise in testingskills, as well as technical skills to help firms successfully implement TestAutomation from the start. LogiGear is one of the few firms that specialize inTest Automation with the depth and breadth of services to ensure your success.LogiGear’s Test Automationsolution coversthe following areas:
• Test Automation strategy consulting * Test Automation tool evaluation and selection * Test Automation scripting and execution using leading tools such asSelenium/Appium, Quick Test UFT, TestComplete and TestArchitect * Automated Omni-channel testing: Web, Mobile, Wearbles, Voice-App, etc. * Test Automation for All Major technolgoies & platforms: cross-browser, WebAPIs/Services, Mobile, Desktop, (.Net, Java, etc.) ERP (SAP, Oracle EBS,Microsoft Dynamics, etc.) To learn more about how our Test Automation experts can help you transformyour process for CD, visit logigear.com today!