Goals of Continuous Integration

If a primary goal of continuous integration is to improve software quality and reduce downstream build and test failures, then perhaps the single most important SCM best practice is to enable automated local developer builds. To this end, developers should be provided with access to a local build environment that mirrors the production build environment, including:

• Database schema
• Configuration files
• Environment variables
• Shared libraries and files
• Known compiler/linker/runtime environments

This minimizes the “it compiled fine on my machine” syndrome that plagues many development efforts. Once the build environment is available, every effort should be made to automate the local build, so that each developer can build easily and quickly without having to perform a series of manual steps.

Ideally, the developers should have access to a private build area that uses the same tools and configurations as the production builds. As part of configuring a local build, developers should be trained to take several steps to ensure that local changes will have minimal negative effects once shared with other developers and teams. These steps are as follows:

• Updating their private workspace to obtain the latest shared configuration code
• Merging conflicting changes from the shared configuration to their private workspace

By ensuring that developers have known, good build environments, and that they have the latest code with all conflicts resolved, there is high probability that the local build will succeed. If the local build is successful, a continuous integration build that includes the developer’s changes should also succeed.

Shifting to Continuous Integration and Frequently Updating Code

Traditionally, developers tend to put off sharing their changes, sometimes for days, because they don’t want to affect other people too early, or don’t want to get blamed for breaking the build. Unfortunately, this strategy tends to backfire and typically leads to more problems in debugging larger sets of changes after a build breaks. Independent of whether continuous integration is being employed, encouraging developers to build, test, and share their code in small “chunks” is a simple and effective way to improve team collaboration and reduce costly broken builds.

The importance of frequently updating the code in the SCM system is amplified when organizations move towards a continuous integration model. Continuous integration represents a paradigm shift for software development, emphasizing communication between developers about what changes have been made. By breaking down tasks into small chunks that take several hours to complete, developers can “commit” or “promote” their changes frequently and receive immediate feedback about the quality of the changes through the continuous integration build and unit test results. As teams get accustomed to this new approach, developers also gain a sense of progress by seeing not only their own changes build successfully, but also seeing the changes that other developers have made available.

Streams are the workhorse of an AccuRev depot. Without streams, everyone would commit code to a single “bucket.” As you know, streams are connected together in a hierarchy that defines a promotion-based workflow. For mainline development, a workflow can be as simple as:

Dev > Test > Prod

or, sophisticated like:

Dev > UnitTest > Int > LoadTest > FlightTest > Prod

The Pattern

While streams are always used for organizing the core workflow, they can also be extended to logically organize developer tasks using a “stream-per-task” paradigm. The concept is simple: create a new stream for every task that one or more developers plan to work on. Name the stream logically, as in “Feature X” or “BugFix Y”. The developers then create their workspaces from this stream. This pattern works exceptionally well with agile development groups where work is broken down into tasks.

The Advantages

In theory, the task stream is just another stream and promotion step on the way to delivering changes to the core workflow (see adjacent picture). Though, in practice, the advantages are numerous:

• Organize development activity by feature or fix
• Visually see all active projects (removed task streams are implicitly finished)
• Visually see and manage who is working on any given project
• Develop individual projects against latest mainline without forced committing to the core (relies on inheritance and significantly reduces rollback)
• Control access to projects by user or group using stream locks
• Retarget entire projects and teams with a single drag-n-drop

The Critics

No suggestion is without its critics. Let’s address a few of the common claims. Claim: adding a new stream creates an unnecessary promotion step. Response: use a pass-through stream instead! Claim: creating a stream-per-task will create 10′s, or 100′s of streams. Response: Yes, but each stream is a 120-byte (yes, byte) record of meta data on the server so space consumption is negligible; the GUI ‘zoom’ feature makes navigating thousands of streams manageable; remove each stream when the project is finished; and using a good naming convention goes a long way. Claim: Creating streams is yet-another-step for someone; who will do it? Response: it depends; Sometimes savvy project managers enter them in AccuRev; other times it’s the release manager(s) who take this on as a routine task; but I say, empower the developers! Why not have the dev leads or project-oriented developers do the task — it’s sooo simple and can be controlled with locks, ACLs, or basic triggers.

For those critics still remaining who need proof, I’ve provided an illustrated stream hierarchy where everyone contributes to a single bucket. Compare this picture with the one above…. and I’m sure you’ll be convinced. In short, the drawbacks of everyone committing to the same bucket include:

• Unable to collaborate in isolation
• Requires reverting (unmerging) features who’s release has been postponed
• Limited visibility to active projects; requires strict wspace naming; hard to enforce
• Retargeting project work requires multiple workspace reparenting
• Unable to control per-project delivery or access with locks or ACLs

Concrete Example

With the pros and cons out of the way, lets show a concrete example. For both discussions above, I intentionally created stream structures in abstract form using generic names and a layout that is comparable. But lets get real. Here is a stream structure that represents a more realistic environment and even includes elements from a previous blog post on using the pass-through stream. Isn’t it great to see how this stream-per-task paradigm helps methodically organize activity across the entire depot!

Are there any other pros or cons to this approach? Does anyone have examples of using this pattern in their own implementation of AccuRev?

/happy tasking/ – dave