There’s no denying that this has certainly been the Agile decade for the software development industry.  It’s evident all around us in this tenth year since the Agile Manifesto was created. Most companies and development organizations today have implemented some form or aspect of Agile methodology into their software development processes. Whether you’re aiming for pure Agile or a mixed/hybrid approach, proven best practices in all phases of the software development lifecycle are crucial to success.

This is especially true in the case of continuous integration, one of the foundational aspects of the Agile methodology. The concept of continuous integration, as defined by Martin Fowler, is “a fully automated and reproducible build, including testing, that runs many times a day.  This allows each developer to integrate daily, thus reducing integration problems.”

With this approach, developers can work more closely in parallel while identify problems and debugging on the fly, accelerating the development process and improving the quality of the finished product.  The benefits of continuous integration are tremendous, but can quickly be eradicated if software configuration management (SCM) best practices are not carefully followed.

There are a handful of SCM best practices that can optimize continuous integration.   Let’s start with a quick look at the first two:

• Using an SCM system to store and version all source code
• Utilizing private developer workspaces

Best Practice: Using an SCM System to Store and Version all Source Code

Parallel development and distributed software teams can make tracking changes a daunting task, especially with the frequent changes that occur when using continuous integration methods.

For this reason, it is important to employ a software configuration management (SCM) system to strictly version changes to the code base. In addition to versioning source code, everything needed to build the system should be placed under version control, including the following:

• Third-party libraries
• Properties files
• Database schema
• Test scripts
• Install scripts

All developers should have at least read-only access to all files needed for the build and should obtain all such files directly from the SCM system. This approach ensures that developers are working with the latest build environment, and is preferable to the common but error-prone practice of placing such files on a shared file server.

To effectively implement continuous integration, all development groups should work from the same central source code repository so that the latest changes from other developers are easily and immediately available.

Best Practice: Utilizing Private Developer Workspaces

In order to fully realize the benefits of continuous integration, software development organizations need to ensure that developers can remain productive regardless of the overall state and stability of the project source code. To achieve this, private workspaces that give developers full SCM capability should be used. Private workspaces enable developers to

• work in isolation
• revert to known “good” states when needed
• checkpoint their changes
• share only mature, well-tested code with other team members

The benefits of isolation are bidirectional—it protects developers from incoming changes, and protects the shared code configuration from incomplete or incorrect changes from any one developer. By creating private workspaces, developers receive all the benefits of SCM for their personal use, including the ability to revert to a previous state, viewing and tracking of changes between software configurations, and setting aside changes to begin work on a different task.

Once a new known good state is reached (for example, when a developer completes engineering and testing work on a feature), developers should checkpoint their work, typically by “checking in” or “keeping” the local changes in the SCM system. The checkpoint ensures that the developer’s work is safe on the SCM server and that the checkpoint can be revisited at any time. However, since the changes have not been shared, other developers and teams are not affected.

When a developer breaks isolation and decides to share a code change, he or she is essentially making an assertion that the change has reached a higher level of maturity. This, coupled with the use of local developer builds, helps to ensure that only mature, well-tested code is passed on to the rest of the development team, a primary benefit of continuous integration.

by Rob Mohr, AccuRev

In one of many travels and customer visits, I came across a very cool way that AccuRev was helping to improve the way development teams do their work. To be more specific, this group was using Change Packages integrated with the JIRA Issue Tracking system to manage changes across their various product releases. They also used CruiseControl for continuous integration that would kick off nightly builds and notify the team with the results of the build.

From what they told me, the success of builds has significantly improved since they started using AccuRev because of the ability for the developers to work in their own private workspaces where they can integrate and unit test before promoting their changes for the rest of the team. Although broken builds are, for the most part, a thing of the past, they will still occur once in a while and need to be fixed ASAP.

Here is how they do it with AccuRev

The stream structure below is a simpler view of their overall software development process, but will be sufficient to show the use case.

Promoting to the Integration Stream

To start, the 4 developers below have made changes in their workspaces that will be promoted and associated to 4 different issues.

As you can see below, the integration stream (EntSoft_Client_Int) is “aware” of which issues are active in the stream. These are the new “change packages” introduced in the stream to be included in the next nightly build.

Build Fails in the Integration Stream

The next morning, the team is notified that last nights build failed via an email notification from CruiseControl. They have also scripted CruiseControl to automatically enable a time based stream called the “Temp_Fix_Build” stream and assign the appropriate transaction number to rollback the change packages from last night.

Assign the Developer to Fix the Build

One of the developers creates a workspace on the Temp_Fix_Build and “change palettes” over each change package one at a time.  This gives them the ability to mix and match change packages together to determine which one of them is the problem.

Problem Solved

After the culprit is fixed, the repaired change package(s) are promoted back into the integration stream for all to share.

For those tracking the movement of luminaries such as Martin Fowler and Kent Beck or looking for scalability advice from the architects at companies like Orbitz, Ebay, and Linked-In… QCon ’07 in downtown San Francisco is the place to be!

The conference is packed with senior architects, software engineers, and open-source contributors galore — over 400 were rumored to be in attendance. With speaker topics ranging from enterprise scalability to Agile practices, the audience was nothing short of being at the top of their game. Huddled together at the entrance to the conference rooms were a number of vendors showing off new warez including AccuRev. Here’s a shot of Damon Poole & Cliff Utstein (top-right), Dave Thomas (left), and John Wall (bottom-right). The AccuRev booth had a constant flow of folks amazed at how the stream-based architecture brings a refreshing approach to managing software configurations and supporting agile practices. We also had some cameo appearances from existing customers like Authorize.Net and Orbitz.com.

Agile development methodologies is a major theme of the conference. For someone looking for advice on agile, just standing in the middle of the exhibit hall is all it takes — everyone is talking about best practices, success stories, and failed attempts.

We had a constant stream of people intrigued by our stream-based architecture and inherent support for agile practices. Here’s a list of common discussion points:

private workspaces: commit-early, commit-often

stream inheritance: merge-early, merge-often and sharing iterations early and automatically with parallel development efforts

issue tracking integration: assign, deliver and track development activity to stories/issues/features

continuous integration: integrations with cruise control, finalbuilder, electric-cloud, and others

refactoring: IDE integrations with eclipse, Intelli-J, Visual Studio support application-wide refactoring with version control

staged workflows: organize distributed teams and isolate integration areas from testing areas for explicit and repeatable access to known configurations.

snapshots: guaranteed reproducibility of labeled configurations for builds known to be ‘good’

reparenting: retarget active development to known good configurations for testing or stable development

If you didn’t have a chance to attend this years QCon, be sure to put next years event on your calendar!

/happy conferencing/ – dave

It’s amazing how many times I’m asked about how you can collaborate on sub-projects or individual features with other developers without affecting the rest of the development team.  As a matter of fact, this came up the other day, which is why I thought it would make a good post.

There are several ways that teams can collaborate in AccuRev, and depending upon what the situation is, would choose different methods.  I describe two simple scenarios where 2 developers are working on a few files together and another where several developers are teamed together to work on a sub-project.

Scenario 1: Two developers are working on the same foo.c file and must complete the work before it can be merged with the rest of the project (ie. promoted).

The great thing about private workspaces is that developers have a place where they can commit their code without impacting the rest of the team.  The code *is* placed on the server, but still remains private to the developer.  Meaning, other developers who update their workspaces will not propagate those changes into their private workspaces.

However, since the code *is* on the server, another developer can explicitly pull that version into their workspace using the “send to workspace” command (aka. “co”).  The developer can make changes and keep the file to place the new version on the server, again stored in the private workspace.  This exchange continues until the work is completed and the file can be promoted to be shared with the rest of the team.

This is typically a good method to use for collaborating on just a few files, but what if you had an entire sub-project to share with many developers.

Scenario 2: Many developers working on entire sub-project which must be completed with testing before merging with the rest of the project.

This scenario is the “bread and butter” of what AccuRev and the stream based architecture is all about.

To set this up, you simply create a stream for the sub-project from the main project stream.  The sub-project development team creates workspaces off the sub-project stream while the rest of the development group is based upon the main project stream.

With inheritance, any changes made by the main project team is automatically inherited by the sub-project team so they [sub-project team] are continuously integrating with the main development effort.  The main project team is completely unaffected by the changes made by the sub-project team.  The sub-project team can also control what changes are inherited into their stream by setting a time basis on their stream.  The time basis will only allow the changes to flow into their stream up to the time set.

Eventhough I described this scenario as a method for larger groups, it can also be used for the simple case of 2 developers working on the same file.   Streams are so architectually light weight, that they can be created and removed in a matter of seconds.

 I hope this little tip helped for those looking to collaborate.

By now you’ve probably heard about workspace reparenting in AccuRev. You know… the fancy drag-n-drop of a workspace from one stream to another. But what’s all the hype? Ultimately, workspace reparenting means that a workspace, on disk, can ‘morph’ into the configuration of it’s parent stream (See video) . So one day you are working on a mainline feature and the next day, with a single command, you can instantly have the configuration from a buggy release 2 months ago on disk, in the same location, with the exact configuration of files at that point in time. Poof! Well… first off, reparenting isn’t magic. You see, each and every stream in AccuRev knows at least two things – (1) its parent stream and (2) any active changes to its own relative configuration. Knowledge of the parent stream is by way of reference, in the same way that an OO subclass knows of its parent class. So if you change the parent reference, you instantly have visibility into the new parent configuration, in combination with any active changes that come along for the ride (as needed, most likely). Remember, workspaces are streams too!

Fun Toy or Power Tool?

The only “fun toy” you’ll find in AccuRev is the easter egg hidden in Help–>About where if you press CapsLock-Pause with your left hand you can play a tetris clone… ok, kidding. Workspace reparenting is a wickedly powerful feature that, when used properly, can save time, energy, and disk space and be a great powertool. So when is reparenting useful?

Retargeting a Delivery – Remember the time when you spent Saturday night cranking out a hot-fix… only for it to be reclassified to a lower priority on Monday and scheduled for next weeks build? No problem! Simply reparent your workspace from the release hotfix stream to the mainline area and promote the changes. done. go home.

Large Code Footprints – Lets say you have a 400Gb footprint of source code for any given release. On a particular day, you just finished committing 174 source files to mainline for a new feature. You then get tasked to the bug squad to tackle 9 high-priority bugs found in the recent production release. Rather than creating a brand new workspace from the old release snapshot, you can simply reparent your ‘mainline’ workspace and run update. You’ll most likely get the old versions of 174 files and probably a few dozen or hundred other files that changed… but the point is, you just get the delta. And waiting for 2G of changes over the wire compared to 400G is a significant time savings for you and others sharing the pipe.

Tracing Defects - You know those bugs that crop up and someone asks, “how long has that runtime defect been around?” Take a single workspace, create a unit test that forces the failure, then reparent to each and every release snapshot in question. Each time you do an update, you’ll get a corresponding configuration of code on disk -and- the tweaked unit test (active changes come along for the ride, remember!). Run the test, verify the failure. Bingo – a recipe to quickly test how far back a defect went. Just keep reparenting and running the test(s) until you find a configuration that works.

Rapid Application Development – Are you responsible for cranking out a few features all at once? Need to hop between them frequently? Sure, you could create a separate workspace for each feature and that’s probably a better practice in most cases… but nothing stops you from using a single workspace and context-switching by reparenting. As long as you keep/promote each feature’s set of files out of the workspace, then reparenting is a breeze. This works well if you follow a stream-per-feature paradigm.

So now you might say, “But isn’t reparenting the same as updating my local work area with a different ‘branch’?” Nope. Don’t forget about the most significant architectural difference between streams and branches… inheritance. We’ll save that for another thread.

One of the great things about workspaces is that they are a super tiny meta-data record in the AccuRev meta-database… just a pointer to parent stream and pointer to host machine and dir on disk. That’s it. Workspaces are cheap entities. Reparenting simply changes the single parent reference reference and updating only retrieves the deltas. Make reparenting part of your routine and the benefits will follow.

/happy reparenting/ – dave