An Introduction to Continuous Integration and Workflows

Once upon a time integrating the various modules of a software project or product was one of the most daunting and challenging tasks. In many cases, software project managers were confronted with the notorious “80% done and 80% remaining” phenomenon: While the implementation of the individual software modules of the project were completed, their integration was extremely challenging. Specifically, software teams had to deal with brittle architectures, modules that did not fit each other, as well as their inability to objectively estimate project progress. Fortunately, this situation has nowadays changed thanks to the introduction and wide use of Continuous Integration (CI) practices as part of software engineering.

What is Continuous Integration?

In a nutshell, Continuous Integration is the practice of integrating all the different modules of a software project several times per day. This usually happens in a shared repository that resides in a server or cloud infrastructure. The process involves code developed by different developers, which is why CI is a practice closely linked to collaborative software development processes.

The idea of CI was originally introduced in the early 90s and evolved considerably ever since. During the last two decades, CI is a mainstream software development principle, which is part of most agile software engineering methodologies. CI is also a core element of the popular DevOps (Development and Operations) software development paradigm, which is currently followed by most innovative enterprises, including both high-tech start-ups and Fortune 500 companies. CI’s popularity lies in that it alleviates a proclaimed misalignment between software developers and operational teams. Specifically, software developers change their code frequently to add more functionalities to their product and refactor previous versions for improvement. On the other hand, business management and the operational teams ask for stable versions of software products that can be deployed and used in production environments. In this context, CI ensures that developers’ changes are smoothly integrated into the product, without any adverse effect on its existing functionalities.

From a technical perspective, CI creates a consistent, automated, and effective pipeline for developing and building software products. The pipeline involves all the different steps of the application development lifecycle, including development, building, testing, and packaging of a software product. Based on such pipelines, developers are encouraged to commit their code changes frequently, which enhances collaboration and software quality. In modern CI environments, each code check-in is followed by an automated build of the software product, which helps teams verify the correctness of the new code and detect potential problems as early as possible.  Overall, CI is grounded on a set of practices that enable software teams to implement changes frequently with the least possible effort, which leads to software products of better quality at the least possible cost.

The establishment of a consistent and effective CI process is a key enabler of Continuous Delivery (CD). This is because CI enables the continuous production and availability of a tested and validated software product. As developers push changes (e.g., new features) in the CI environment, a new version of the entire software product is produced. Therefore, the terms CI and CD are commonly used together (i.e. CI/CD process) in the scope of software engineering and DevOps contexts. Moreover, continuous testing is an integral element of CI/CD practices as integrated and delivered products must be properly tested.

The Steps of a CI Pipeline

In practice, CI/CD pipelines include more steps that simply check-in code, building, and testing the software product. To this end, developers harness the build, testing, and version control functionalities of specialized servers that provide CI related functionalities (i.e. CI servers). Specifically, a typical CI/CD workflow involves:

• Pulling code from the version control system and building it. As part of the build process software components, database scripts, front end modules, and other elements of the software product are automatically built.
• Configuring the target infrastructure where the code shall be deployed (e.g., on-premise server or cloud computing infrastructure) and moving the code to it.
• Dispatching the various modules that comprise the software product to their target environments, while linking them to appropriate API services and databases.
• (Re)starting servers and other elements of the target operational infrastructures such as web servers, database servers, and test servers.
• Executing continuous and automated tests to ensure that the new code does not have any adverse effect on the product and does not create any problems in the deployment environment. A variety of testing techniques are usually employed such as automated regression and performance tests.
• Indicating the proper rollback actions and configuring the rollback environment in case some tests fail. Note however that if all tested are passed successfully, there is no need for rollback.
• Notifying developers and administrators about the status of the product and of the delivery, while enabling them to access relevant log files and status reports. Sophisticated CI tools include advanced dashboards and reporting tools, which facilitate developers to track errors and trace back to their root causes.

As already outlined, the above-listed steps are greatly facilitated by CI servers, which automate and manage entire CI/CD pipelines. CI servers facilitate the configuration and management of all the artifacts of the CI/CD processes, including for example environment variables, deployment options, security certificates, built configurations, test configurations, as well as error messages and notifications. Furthermore, these tools undertake to package these artifacts as needed towards a successful software product delivery.

Advanced CI Practices

Nowadays, competent software teams tend to employ advanced CI practices beyond simple pipelines. For instance, they carefully plan the timescales where they implement changes and fixes while controlling the frequency of new code uploads and the subsequent execution of tests. Moreover, they usually work with many different environments where their code is integrated, including development, testing, and production environments. CI/CD pipelines push code changes to these environments to ensure that code is adequately tested prior to moving it to production. Furthermore, CI/CD processes take care of additional steps, which ensure that the product is up to date. Such steps include integrating the latest third-party libraries and patches, as well as ensuring that the product is synchronized to the most up to date data.

In recent years more advanced CI/CD features have emerged. For instance, there are configuration mechanisms that enable teams to dynamically define the features and code that will be tested and moved to production. Likewise, some teams implement advanced version control functionalities, which define when each piece of new code will be tested and integrated into the production code. In this way, it is possible to distinguish features that should make it to the production in short time scales from features that require longer development cycles. As another example, it is possible to automate changes to multiple delivery environments. Furthermore, CI/CD processes are evolving to support novel concepts in cloud computing and workload management, such as pipelines involving Kubernetes and the Function-as-a-Service paradigm.

Overall, CI/CD is the present and future of software engineering. Software development enterprises must embrace, support, and fully leverage the benefits of CI/CD. Moreover, they should keep an eye on its continuous evolution towards supporting novel software architectures such as serverless computing. If you are not on CI/CD, it’s time you adopt it. On the other hand, if you are already using CI/CD you should keep up with its evolution.