Understanding DevOps

Continuous Testing and Continuous Monitoring

What is the goal of Continuous Integration?

Is it to enable Continuous Delivery of the code developers’ produce out to users? Yes, eventually. But first and foremost it is to enable ongoing test and verification of the code. It is to validate that the code produced and integrated with that from other developers and with other components of the application, functions and performs as designed. Once the application has been deployed to a production system, it is also a goal to monitor that application to ensure that it functions and performs as designed in a production environment, as it is being used by end-users.

Logigear_Understanding DevOps

This brings us to the two practices of DevOps that are enabled by Continuous Integration and Continuous Delivery.

Namely:

  • Continuous Testing
  • Continuous Monitoring

Continuous Integration and Delivery are both (almost) meaningless without Continuous Test. Not having motoring and hence, not knowing how the application is performing in production makes the whole process of DevOps moot. What good is having an streamlined Continuous Delivery process if the only way you find out about your applications’ functionality or performance being below par is via a ticket opened by a disgruntled user?

Let’s break this down.

Continuous Integration and Continuous Delivery facilitate Continuous Testing

Individual developers work to create code – to fix defects, add new features, enhance features, or to make the code perform faster – could be one of several tasks (work items) they could be working on. When done, they deliver their code and integrate it with the work done by other developers on their team, and with unchanged code their team owns. They would along the way run Unit Tests on their own code.

Once the integration is done, they would do Unit Tests on the integrated code. They may run other tests such as White box Security tests, code performance tests, etc. This work would then be delivered to the common Integration area of the team of teams – integrating the work of all the teams working on the project and all the code components that make up the service, application or system being developed.

This is the essence of the process of Continuous Integration. What makes this process continuous is where the individual developers’ code is integrated with that of their team as and when they check-in the code and it gets delivered for Integration. (For more detail, read Part 2 of this Understanding DevOps series). The important point to note here is the goal of the Continuous Integration process – to validate that the code integrates at all levels without error and that all tests run by developers run without error. Continuous Testing starts right with the developers.

After validating that the complete application (or service or system) is built without error, the application is delivered to the QA area. This delivering of code from the Dev or development environment to the QA environment is the first major step of Continuous Delivery.

There is Continuous Delivery happening as the developers’ deliver their code to their teams’ integration space and to the projects integration space, but this is limited to being within the Dev space. There is no new environment to target. When delivering to QA, we are speaking of a complete transition from one environment to another. QA would have its own environment on which to run its suites of functional and performance tests. DevOps principles preach that this environment be production-like.

In addition QA would potentially also need new data sets for each run of the suites of tests it runs. This may be even one or more times every day as Continuous Integration leads to Continuous Delivery at a steady stream. This means that the Continuous Delivery process would not only require the processes to transition the code from Dev to QA, but to also ‘refresh’ or provision new instances of QA’s ‘Production-like’ environment, complete with the right configurations, and associated test data to run the tests against.

This makes Continuous Delivery a more complex process than just ftp-ing code over. (I will discuss this in more detail in a later post of Infrastructure automation and Infrastructure as code). The key point to note is that the goal of Continuous Delivery is to get the code ready for test. To get the application to the right environment – continuously, so that it can be tested – continuously.

If one extends the process described above to delivering the service, application or system to a staging and eventually a production environment, the process and goal remains the same. The Ops team wants to run their own set of smoke tests, acceptance tests and system stability tests before they deliver the application to the ‘must-stay-up-at-all-costs’ production environment. That is done using the Staging environment. This is a production-like environment that needs to be provisioned just like the QA Environment. It needs to have the necessary scripts and test data for acceptance and performance tests that Ops would run. Only when this last phase of Continuous Testing is complete would the application be delivered to Production. Continuous Delivery processes would perform these tasks too – provision staging and production environments and deliver the application.

Continuous Monitoring

In Production, the Ops team manages and ensures that the application is performing as desired and the environment is stable via Continuous Monitoring. While the Ops teams have their own tools to monitor their environments and systems, DevOps principles suggest that they also monitor the applications. They need to ensure that the applications are performing at optimal levels – down to levels lower than system monitoring tools would allow. This requires that Ops teams use tools that can monitor application performance and issues. It may also require that they work with Dev to build self-monitoring or analytics gathering capabilities right into the applications being built. This would allow for true end-to-end monitoring – continuously.

The Four Continuous Processes of DevOps:

It would not be much of a stretch to say that the practice of DevOps is really up of the implementation of these four basic processes:

  • Continuous Integration
  • Continuous Delivery
  • Continuous Testing
  • Continuous Monitoring

Chart 1

Of these, the end goal is really Continuous Testing and Continuous Monitoring. Continuous Integration and Continuous Delivery get the code in the right state and in the right environment to enable these two processes. Testing and Monitoring are what will validate that you have built the right application, which functions and performs as designed. In future posts, I will go into more detail of the implementation of these four processes and the associated practices that enable them.

Sanjeev Sharma
Sanjeev is an IBM Distinguished Engineer, and the CTO for DevOps Technical Sales and Adoption at IBM, leading the DevOps practice across IBM. Sanjeev is responsible for leading the worldwide technical sales community for DevOps offerings across IBM’s portfolio of tools and services, working with IBM’s customers to develop DevOps solution architectures for these offerings, and for driving DevOps adoption. He speaks regularly at conferences and has developed DevOps IP, including the DevOps For Dummies book published in 2013.

Sanjeev is a 20 year veteran of the software industry. Sanjeev has an Electrical Engineering degree from The National Institute of Technology, Kurukshetra, India and a Masters in Computer Science from Villanova University, United States. He is passionate about his family, travel, reading, Science Fiction movies and Airline Miles. He blogs about DevOps at http://bit.ly/sdarchitect and tweets as @sd_architect

The Related Post

By Jez Humble and David Farley Continuous Delivery from Jez Humble and David Farley is an important contribution to the field of software development. It takes continuous integration to the logical conclusion and covers how to set up a continuous integration system, delving into everything from check-in to delivery to production. It doesn’t state you ...
LogiGear Magazine – December 2013 – Cloud Testing
It is a fundamental role for testing teams to align their test design, test automation, and test case development with DevOps–not only to verify that code changes work but that the changes do not break the product. A key differentiator of DevOps is testing maturity. An organization can automate their integration, testing, delivery, and monitor, ...
Fitting QA into a modern DevOps group In a traditional software engineering organization, the QA group is often seen as separate from the Development group. Developers and testers have different roles, different responsibilities, different job descriptions, and different management. They are two distinct entities. However, for folks outside the engineering team – say in Operations ...
LogiGear Magazine June Testing in Continuous Delivery Issue 2017
…On what you need to know before making the transition to EaaS 1. What are the main differences between cloud-based environments and cloud infrastructure? An environment is a collection of infrastructure elements working in conjunction to enable an application stack to work. For example, a simple 3-tier application, with a web front-end component, a business logic ...
How to Adopt the “Third Way” in the Dojo’s Method to Master CD In The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations the authors describe “The Three Ways” – the underlying principles forming the basis for all DevOps practices. 
LogiGear University announces the launch of a new, free video series on Testing in DevOps and Continuous Testing which is available today.
For this month’s book review, I read Continuous Testing for DevOps Professionals: A Practical Guide from Industry Experts, by various authors and edited by Eran Kinsbruner. The book is divided into 4 sections: Fundamentals of Continuous Testing, Continuous Testing for Web Apps, Continuous Testing for Mobile Apps, and The Future of Continuous Testing. The Fundamentals ...
Introduction Everything changes. It’s the only constant. The landscape of software testing is undergoing a fast and dramatic change driven by societal, business and technology changes that have placed software everywhere. Computing is ubiquitous. There is hardly anything today that doesn’t contain a CPU or information transmission capability, with software to drive it. From smart toasters ...
DevOps has been described as Agile on Steroids; DevOps has also been described as Agile for Operations/IT. I like both of those descriptions. Many organizations want Development, Test, and Operations teams to move to DevOps now. DevOps is a big topic, but DevOps is not the focus of this article. We will not be talking ...
  LogiGear_Magazine_June 2016_Testing in the New World of DevOps  

Leave a Reply

Your email address will not be published. Required fields are marked *

Stay in the loop with the lastest
software testing news

Subscribe