Continuous Integration

Bob	I just read a great book titled “Getting Things Done” by David Allen.
Mark	Is it really any different from any of the other gazillion self-help books that are out there?
Bob	It seems to be working for me. I was a big fan of the To-do list and I had a massive list that I regularly prioritized and kept on my PocketPC. The only problem was that I never seemed to get done with the things on my list.
Mark	Maybe the book will help me get things done with one of my clients. I just spent a frustrating week fighting build issues and not making any forward progress on adding new features to the project.
Bob	Allen's book probably won't help much with build issues, but perhaps I can. Give me some details.
Mark	It seemed like whenever anyone had code to integrate with the rest of the team, they'd commit it to CVS and then we'd discover that the build was broken. Then everybody would have to diff what was in CVS with the code that they were working on, get it to build, check it in, and the build would break again. It's an endless loop of frustration.
Bob	How often do team members integrate the code that they're working on?
Mark	When they get done with it.
Bob	The question was, "How long?" You know, a duration, a span of time; on average.
Mark	We try to break our activities into small pieces so people check in every few days.
Bob	I think that's the first problem. I try to get my clients to check in several times per day.
Mark	That's just nuts. We wouldn't do anything but diff, merge, and check in again.
Bob	Isn't that what you said you did all last week?
Mark	Well, yeah. But I sure don't want to do it even more often. It was a painful process.
Bob	''I've found that if something hurts and I do it more often, it doesn't hurt as much.
Mark	That doesn't make sense.
Bob	Sure it does. Think back to when I started bike riding with you but I could only ride once every few weeks because of my hectic travel schedule. The day after a ride, getting out of bed was a new experience in pain. But when we started riding every few days, it wasn't painful at all.
Mark	What does that have to do with software and my build issues?
Bob	Why is committing code so painful?
Mark	Because of the never-ending diff and merge.
Bob	Exactly. And diff and merge are painful because there is a lot to diff and merge - because you've waited so long since the last diff. What if you checked in your code changes whenever your code was in a stable state?
Mark	You seem to be making a distinction between being done with my piece of the project and what you're calling a stable state.
Bob	Yes. A stable state is when you can build and all of your tests are passing. I claim this can happen several, if not many, times per day.
Mark	Not all that much code changes in just a few hours.
Bob	My point exactly. Not much changed so diffing is minimized if not eliminated altogether.
Mark	But this would only solve half the problem - the diffing half. How would this help the problem of the broken build?
Bob	Here's the procedure we try to get our clients to adopt. The more often you do it, the less time it takes. 1) Do a CVS update. This will bring down any new code to your workstation and point out files that you've changed that your coworkers have also changed. There shouldn't be much if you do this every chance you get. 2) Delete all of your .obj's or .class files, depending on your language and do a complete build on your workstation. 3) Run all of your unit tests (you are writing unit tests aren't you?). They must all pass. 4) Run all of the automated customer acceptance tests. All of them that are supposed to pass, should pass. 5) Do a CVS commit.
Mark	That sounds like a good plan, which is better than what we have now, which is essentially no plan at all.
Bob	Good luck.

---

!commentForm

Add Comments Below:

Chris, 29 Jan 2005

How well have you seen this scale with:
1. Lines of code? - if you have a lot of code to rebuild it slows down a single cvs update/commit cycle
2. Number of developers? - might depend if you advocate serialization of commits - e.g. do you have a token system so that one developer must update/commit at one time, as if so then a 10 to 20 person dev team will be restrictive. (And if you dont advocate tokens, then that is another question... :-)

---

We're doing this on a 500K NCSL (1M SL) project, with about 30 developers in 4 distinct geographical locations. It's working well.

We're using CruiseControl to support it, and we've divided the project into about 25 sub-projects. A commit causes any affected sub-projects to build incrementally, and we also run a daily clean build. Since the daily clean build takes about 70 minutes at the moment, we don't want to run that every time someone checks in changes. The sub-projects typically take 2-5 minutes to build, so the developer gets the "build successful"/"build broken" email within 5 to 8 minutes.

An additional advantage is that we've got two of these build servers set up, one on Windows and the other on Linux: we can build our JNI code for each platform automatically, and run unit tests on each platform automatically. A developer working on their laptop can get quick feedback for the other platform.

We aren't using any "token system" for managing commits (we're using Perforce instead of CVS, but the ideas are the same). Instead, we expect the developers to merge their changes with the current state of the code, and then test, before they check in. If you've got a large checkin on a busy day, you may need to merge more than once; but that's a good argument for making small incremental changes.

Leif Bennett
8 Aug 2005