Understanding DevOps

Continuous Testing and Continuous Monitoring

What is the goal of Continuous Integration?

Is it to enable Continuous Delivery of the code developers’ produce out to users? Yes, eventually. But first and foremost it is to enable ongoing test and verification of the code. It is to validate that the code produced and integrated with that from other developers and with other components of the application, functions and performs as designed. Once the application has been deployed to a production system, it is also a goal to monitor that application to ensure that it functions and performs as designed in a production environment, as it is being used by end-users.

Logigear_Understanding DevOps

This brings us to the two practices of DevOps that are enabled by Continuous Integration and Continuous Delivery.

Namely:

  • Continuous Testing
  • Continuous Monitoring

Continuous Integration and Delivery are both (almost) meaningless without Continuous Test. Not having motoring and hence, not knowing how the application is performing in production makes the whole process of DevOps moot. What good is having an streamlined Continuous Delivery process if the only way you find out about your applications’ functionality or performance being below par is via a ticket opened by a disgruntled user?

Let’s break this down.

Continuous Integration and Continuous Delivery facilitate Continuous Testing

Individual developers work to create code – to fix defects, add new features, enhance features, or to make the code perform faster – could be one of several tasks (work items) they could be working on. When done, they deliver their code and integrate it with the work done by other developers on their team, and with unchanged code their team owns. They would along the way run Unit Tests on their own code.

Once the integration is done, they would do Unit Tests on the integrated code. They may run other tests such as White box Security tests, code performance tests, etc. This work would then be delivered to the common Integration area of the team of teams – integrating the work of all the teams working on the project and all the code components that make up the service, application or system being developed.

This is the essence of the process of Continuous Integration. What makes this process continuous is where the individual developers’ code is integrated with that of their team as and when they check-in the code and it gets delivered for Integration. (For more detail, read Part 2 of this Understanding DevOps series). The important point to note here is the goal of the Continuous Integration process – to validate that the code integrates at all levels without error and that all tests run by developers run without error. Continuous Testing starts right with the developers.

After validating that the complete application (or service or system) is built without error, the application is delivered to the QA area. This delivering of code from the Dev or development environment to the QA environment is the first major step of Continuous Delivery.

There is Continuous Delivery happening as the developers’ deliver their code to their teams’ integration space and to the projects integration space, but this is limited to being within the Dev space. There is no new environment to target. When delivering to QA, we are speaking of a complete transition from one environment to another. QA would have its own environment on which to run its suites of functional and performance tests. DevOps principles preach that this environment be production-like.

In addition QA would potentially also need new data sets for each run of the suites of tests it runs. This may be even one or more times every day as Continuous Integration leads to Continuous Delivery at a steady stream. This means that the Continuous Delivery process would not only require the processes to transition the code from Dev to QA, but to also ‘refresh’ or provision new instances of QA’s ‘Production-like’ environment, complete with the right configurations, and associated test data to run the tests against.

This makes Continuous Delivery a more complex process than just ftp-ing code over. (I will discuss this in more detail in a later post of Infrastructure automation and Infrastructure as code). The key point to note is that the goal of Continuous Delivery is to get the code ready for test. To get the application to the right environment – continuously, so that it can be tested – continuously.

If one extends the process described above to delivering the service, application or system to a staging and eventually a production environment, the process and goal remains the same. The Ops team wants to run their own set of smoke tests, acceptance tests and system stability tests before they deliver the application to the ‘must-stay-up-at-all-costs’ production environment. That is done using the Staging environment. This is a production-like environment that needs to be provisioned just like the QA Environment. It needs to have the necessary scripts and test data for acceptance and performance tests that Ops would run. Only when this last phase of Continuous Testing is complete would the application be delivered to Production. Continuous Delivery processes would perform these tasks too – provision staging and production environments and deliver the application.

Continuous Monitoring

In Production, the Ops team manages and ensures that the application is performing as desired and the environment is stable via Continuous Monitoring. While the Ops teams have their own tools to monitor their environments and systems, DevOps principles suggest that they also monitor the applications. They need to ensure that the applications are performing at optimal levels – down to levels lower than system monitoring tools would allow. This requires that Ops teams use tools that can monitor application performance and issues. It may also require that they work with Dev to build self-monitoring or analytics gathering capabilities right into the applications being built. This would allow for true end-to-end monitoring – continuously.

The Four Continuous Processes of DevOps:

It would not be much of a stretch to say that the practice of DevOps is really up of the implementation of these four basic processes:

  • Continuous Integration
  • Continuous Delivery
  • Continuous Testing
  • Continuous Monitoring

Chart 1

Of these, the end goal is really Continuous Testing and Continuous Monitoring. Continuous Integration and Continuous Delivery get the code in the right state and in the right environment to enable these two processes. Testing and Monitoring are what will validate that you have built the right application, which functions and performs as designed. In future posts, I will go into more detail of the implementation of these four processes and the associated practices that enable them.

Sanjeev Sharma
Sanjeev is an IBM Distinguished Engineer, and the CTO for DevOps Technical Sales and Adoption at IBM, leading the DevOps practice across IBM. Sanjeev is responsible for leading the worldwide technical sales community for DevOps offerings across IBM’s portfolio of tools and services, working with IBM’s customers to develop DevOps solution architectures for these offerings, and for driving DevOps adoption. He speaks regularly at conferences and has developed DevOps IP, including the DevOps For Dummies book published in 2013.

Sanjeev is a 20 year veteran of the software industry. Sanjeev has an Electrical Engineering degree from The National Institute of Technology, Kurukshetra, India and a Masters in Computer Science from Villanova University, United States. He is passionate about his family, travel, reading, Science Fiction movies and Airline Miles. He blogs about DevOps at http://bit.ly/sdarchitect and tweets as @sd_architect

The Related Post

The book is an incredibly effective and valuable guide that details the risks that arise when deploying cloud solutions. More importantly, it provides details on how to test cloud services, to ensure that the proposed cloud service will work as described. It is a great start to the topic. The 6 chapters detail a paradigm ...
How Halliburton leveraged outsourcing to achieve their goals. Organizations are focusing on speed, both in Continuous Integration and rapid deployment as a competitive advantage. Many software development organizations can significantly shorten development cycles by implementing one or a combination of Agile practices, continuous integration & deployment methods, and feature branches. While these frameworks and techniques ...
By Jez Humble and David Farley Continuous Delivery from Jez Humble and David Farley is an important contribution to the field of software development. It takes continuous integration to the logical conclusion and covers how to set up a continuous integration system, delving into everything from check-in to delivery to production. It doesn’t state you ...
Having the right skills and experience, even if you have to go outside, is essential for designing tests for large-scale cloud deployments. Moving existing applications to a cloud environment adds new dimensions to testing. One of the primary reasons for moving to the cloud is scalability. Capacity to handle traffic and data transfer can be ...
    Eric Minick is internationally recognized as a leading authority on continuous delivery and DevOps. Eric joined IBM four years ago with the acquisition of UrbanCode where he had worked as a developer, technical seller, and evangelist for a decade. Today, he has responsibility for leading the product management team overseeing continuous delivery solutions ...
Times have changed, the tools have improved, and with books like this available you have no reason to not give CI a go. I still remember the first time I was on a project that used NAnt and CruiseControl.NET. It was years ago and both were new tools with plenty of bugs. The project manager ...
A test team’s job is to report test results, not set or guarantee that you will meet the SLAs. In the rush to cloud services, with everything-as-a-service, you will hear people talking about SLAs. What is this about and what does it have to do with testing? A Service Level Agreement, or SLA, is a ...
Do you want to speed up your automated tests by a factor of 10 and deploy your application continuously? In this article we share how the JIRA development team at Atlassian has accomplished this using Stages in Bamboo. Stages have allowed the JIRA Development team to take a week’s worth of testing and condense it ...
From the culture shift, to differences in Agile, Dave Farley and Michael Hackett discuss the nitty gritty of Testing in DevOps. For this issue of LogiGear Magazine, our very own Michael Hackett sat down with one of the godfathers of Continuous Delivery, David Farley. In this exclusive interview, David discusses how test teams and automation ...
Sauce Lab’s Perspective on Integration with Continuous Testing The term ‘DevOps’ implies that implementing an effective development and production workflow is as easy as developing the collaborative interaction between developers and IT operations. On the surface, DevOps may appear as a simple and straightforward idea but, as you will find out, there is more than meets ...
If you haven’t already caught a glimpse, we announced in January the Testing in DevOps and Continuous Testing videos series, available on YouTube!
LogiGear Magazine June Issue 2018: TESTING in DEVOPS

Leave a Reply

Your email address will not be published. Required fields are marked *

Stay in the loop with the lastest
software testing news

Subscribe