Continuously Sporadic vs Continuously Reactive Integration

The Continuous Integration (CI) concept has been around for well over 15 years now. Agile teams have been successfully implementing this approach with CruiseControl, Build Forge, Jenkins, Bamboo, and other CI software. The idea is great, but the popular implementations have a major flaw – the integration isn’t truly continuous. I would call them Continuous Sporadic Integration systems. Let me explain, and share an alternative approach – Reactive CI.

​A typical CI system performs two functions. It periodically polls the source control repository and determines if new code changes have been committed for a specific project. When new commits are detected, it runs a build script to ensure that the new code compiles, passes unit and integration tests, complies with the coding standards, or doesn’t break the build in any other way. The code changes that caused the build to run are captured, so if the build fails, it is obvious what changes broke it. Since source control systems record which user committed each change, it’s also possible to quickly identify the developer who introduced the build breaking code and have them fix it as soon as possible. The goal is to integrate small changes, one at a time. Ideally, if change is atomic, which automatically means authored by a single developer, problems can be identified and addressed as soon as they occur. This leads to high development velocity while the codebase remains working at all times. Sounds awesome! Except, of course, we don’t live in an ideal world. While in many cases this schema works like a charm, in many others it doesn’t.

​Let me state this right off the bat: The CI systems widely used in the industry today are only pseudo-continuous. A couple of conditions must be met for the process to seem continuous:

1. The source control system is being queried for code changes frequently enough that it creates an illusion that the new code is intercepted as soon as it is committed.
2. A query that lists the code commits, if any, returns quickly, virtually instantaneously.
3. The rate of code commits is low compared to the frequency of the polling by the CI system, so a build triggered by a new commit has time to finish before the next commit is observed.
4. Commits are atomic with quiet periods in between two consecutive commits, i.e. the logic of determining the correct timing for a build is reasonably simple.

With a small, modest-sized project, satisfying these requirements is not hard. However, for larger projects that contain hundreds or thousands of files, it is increasingly more difficult.

1. As the size of a project grows, it takes longer to query for new changes. If the query time becomes comparable to the time between two consecutive polls, the polling frequency will need to be decreased. Otherwise, both the CI system and source repository may start wasting computing resources on the overhead of monitoring code changes rather than doing the actual CI. Remember that a lot of time, the query comes back negative – no new changes since the last poll. Speaking of which…
2. At some point, every query starts returning positive, i.e. new code is constantly committed to the repository. There is no more quiet period between commits to kick off a build. There is no longer an easy way to find a good time to run a build, but since we need to keep running builds, chances of a build starting in the middle of a commit, or multiple commits that logically are one change set, increases dramatically. This will cause builds to break more often. We can still wait for a quiet period, but it may be a while. Let’s say the flood of commits happens during certain hours of the day when developers deliver their work. In this case, the integration won’t happen incrementally and it won’t be easy to figure out which change broke a build or who modified that code. The integration won’t be continuous either.
3. It is also logical to assume that the larger the project, the greater number of code contributors get involved. More developers mean a greater chance that a scenario described above will occur. Large projects usually have longer builds too.

The bottom line is: the CI tools currently available work well on small and medium-size projects and teams. Implementing CI for large-scale projects using these tools may be more than challenging because of the underlying sporadic polling schema.

A Solution: Reactive Continuous IntegrationThe solution to this dilemma could be very simple in principal. What if instead of having the CI system poll the source code repository for changes, the source code repository notified the CI system when a build should be run based on the new code being committed? At HCL we call it Reactive Continuous Integration. The implementation of the add-on Reactive CI service for ClearCase is currently under development and will be available later this year. Here is how it works:

1. Every source code change checked into the source code repository generates an event.
2. Events are processed by the rule engine that determines which builds are affected. Remember that the same code change may trigger multiple project builds. For example, a change to a shared library or service.
3. A build is dispatched to the CI system. Alternatively, the CI can still poll, but it is limited to getting the binary “build, no build” decision from the ClearCase Reactive CI service via a REST API. A poll like this is virtually instantaneous since the logic of determining whether or not to build is now the responsibility of the source repository, reactive and pre-made for the CI system to consume.

​This approach closes two major gaps in the current CI implementations:

1. It is truly continuous because build can be triggered immediately following the new code commit. There is no time lag.
2. There are no more code modification queries that return negative, and no wasted CPU cycles and network bandwidth to check for new commits.

Reactive CI makes it possible for the large system development projects to implement Continuous Integration instead of being stuck with the nightly builds. And with it, the system development organizations can become more Agile and adoptive of DevOps.
Continuous Integration is the approach that allows software and system development organization to accelerate innovation and delivery of products and services to market. What we design and build constantly changes, so the methodology and tools must evolve along the way. Continuous Integration is no exception.

How do you achieve CI in your organization? Please share your challenges and success stories below. I look forward to your thoughts and comments.

Alex Seriy
ClearCase/ClearQuest Technical Specialist
alex.seriy@hcl.com
https://www.linkedin.com/in/alexseriy
https://twitter.com/AlexSeriy