Introduction
Continuous integration is a practice that helps developers deliver better software in a more reliable and predictable manner.
This article deals with the problems developers face while writing, testing and delivering software to end users. Through exploring continuous integration, we will cover how we can overcome these issues.
The Problem
First, we will take a look at the source of the problem, which lies in the software development cycle. Next, we will cover some of the change conflicts that can take place during that process, and finally we will explore the main factors that can make these problems escalate, followed by an explanation of how continuous integration solves these issues.
The Source of the Problem
Let’s take a look at what a traditional software development cycle looks like. Each developer gets a copy of the code from the central repository. The starting point is usually the latest stable version of the application. All developers begin at the same starting point, and work on adding a new feature or fixing a bug.
Each developer makes progress by working on their own or in a team. They add or change classes, methods and functions, shaping the code to meet their needs, and eventually they complete the task they were assigned to do.
Meanwhile, the other developers and teams continue working on their own tasks, changing the code or adding new code, solving the problems they have been assigned.
If we take a step back and look at the big picture, i.e. the entire project, we can see that all developers working on a project are changing the context for the other developers as they are working on the source code.
As teams finish their tasks, they copy their code to the central repository. There are two scenarios that can take place at this point.
- The code in the central repository is unchanged
The code is the same as the initial copy. If this is the case, things are simple, because the system is unchanged. All the ideas we had about the system still stand.This is always the case if you are the only developer working on the application and if you have finished your work before the other members of your team. Either way, things are looking good for you. The system you have created and tested can be delivered to users without additional changes.
- The code in the central repository has changed
The second scenario is that the application you have been working on has changed, and you discover this at the point when you try to copy your code over to the central repository. Changes in the code may or may not be in conflict with the ones you’ve made.If there are conflicts, you need to resolve them in order to be able to successfully deliver your code to the users. In this case, things could get complicated.
Next, we’ll explore the types of conflicts that can happen and what you may need to do to resolve them.
Change Conflicts
There are several types of change conflicts that can occur when integrating code. Here are some of the most common ones. We’ll start with the simplest scenarios, and gradually explore the more complex ones.
- The implementation details have changed – You refactored a method, but so did the developer that has already integrated their code into the central repository. The behavior of the method is the same in all three implementations. You will need to pick the version that will stay, and remove the other implementations. You can even come up with a fourth implementation. This is a simple type of conflict, which you can usually resolve within a few minutes.
- The APIs you have been relying on have changed – For instance, the behavior of a certain method has changed. This could affect your code in a number of ways — from minor changes that you might need to make, to major structural changes. There is no silver bullet in such cases. You will need to carefully study the changes and make all the fixes.
- An entire subsystem of the application behaves in a different way – in such cases you will almost certainly be facing a partial, if not a full rewrite of your solution. If this is the case, you will probably need to speak with all the developers working on the application, because such a significant change should not happen without letting the rest of the team know about it.
These and a number of other issues could come up, caused by various factors. Different versions of frameworks, libraries, databases are another potential source of conflicts.
Once you have updated your code so it can be compiled or interpreted, you also need to remember to repeat all the tests that you have previously ran.
These examples show that the amount of work needed to solve a problem that was initially assigned to a developer can easily double.
Escalating Factors
Here are some of the main factors that can make these problems escalate.
- The size of the team working on the project. The number of changes that are being pushed back into the main repository is proportional to the number of people on the project. This makes the process of integrating code into the main repository significantly harder.
- The amount of time passed since the developer got the latest version of the code from the central repository. As time passes, other people working on the same project are integrating more and more of their work, and changing the context in which your code needs to run. Sometimes the changes in the main repository are so big that it’s easier to do a complete rewrite of your solution.
A large number of changes in the system make integration events more complex and can have a huge effect on the productivity of the team. Such situations are even referred to as “integration hell”.
This process has a number of other negative consequences for your business. Testing and fixing bugs can take forever. Your releases are running late. Teams are stressed out because of long and unpredictable release cycles, and morale deteriorates.
Solution: Integrate Continuously
The solution to the problem of managing a large number of changes in big integration events is conceptually simple. We need to split these big integration events into much smaller integration events. This way, developers need to deal with a much smaller number of changes, which are easier to understand and manage. To keep integration events small and easily manageable, we need them to happen often. A couple of times a day is ideal. The practice of doing small integrations often is called Continuous Integration.
The idea is simple, but at the same time it often appears to be impossible to implement in practice. This is because changing the process requires us to change some of our own habits, and changing habits is difficult.
The Practice of Continuous Integration
In order to avoid the previously described issues, developers need to integrate their partially complete work back into the main repository on a daily basis, or even a couple of times a day. To accomplish this, they first need to pull in all the changes added to the main repository while they were working on the code. They also must make sure that their code will work once it is integrated into the main repository. The only way to ensure this is to test every feature of the application.
What first comes into mind when we start considering continuous integration is that the developers would need to spend half of their time every day testing the code in order not to break the code in the main repository for everyone else.
This is why the prerequisite for continuous integration is having an automated test suite. Automated tests take away the burden of the manual, repetitive, and error-prone testing process from the developers. They also make the entire testing process much quicker. A computer can replace hours of manual testing with just minutes of automated testing. Behavior-driven and test-driven development are techniques that help developers write clean, maintainable code while writing tests at the same time. Testing techniques are out of the scope of this article, and you can read more about them in the other articles.
Tests make sense only if they are executed every time the source code changes, without exception. A continuous integration service such as Semaphore is a tool which can automate this process by monitoring the central code repository and running tests on every change in the source code. Apart from running tests, they also collect test results and communicate those results to the entire team working on the project.
The result of continuous integration is so important that many teams have a rule to stop working on their current task if the version in the central repository is broken. They join the team which is working on fixing the code until tests are passing again. The role of a continuous integration service is to improve the communication between developers by communicating the status of a project’s source code.
How to Adopt Continuous Integration
Continuous integration as a practice makes a big contribution to improving the development process, but also calls for essential changes in the everyday development routine. Adopting it comes with challenges that are easy to overcome if the process is introduced gradually.
One of the biggest challenges teams face is the lack of an automated testing suite. A good recipe for overcoming this situation is to start adding automated tests for all new features as they are being developed. At the same time, the developer working on a bug fix should also work to cover the related code with tests. Whenever a bug is reported, the team should first write a failing test to demonstrate the existence of bug. Once the fix is created, the tests should pass.
Over time, the automated tests suite gradually becomes more comprehensive, and the developers begin relying on it more and more. Adopting a continuous integration service to communicate the status of the tests to the entire team in the early stages of a project is also important, because it raises awareness of the project status among team members.
Conclusion
Introducing continuous integration and automated testing into the development process changes the way software is developed from the ground up. It requires effort from all team members, and a cultural shift in the organization.
Big changes in the workflow are not easy to pull off quickly. Changes have to be introduced gradually, and all team members and stakeholders need to be on board with the idea. Educating team members about the practice of continuous integration practice and building the automated tests suite needs to be done systematically. Once the first steps have been taken, the process usually continues on its own, as both developers and stakeholders begin seeing the benefits of automated testing suites and the peace of mind that this practice brings to the entire team.
P.S. Would you like to learn how to build sustainable Rails apps and ship more often? We’ve recently published an ebook covering just that — “Rails Testing Handbook”. Learn more and download a free copy.