In the last few posts I have discussed some of, what I consider to be, the most valuable Agile methods for development.  The list is pretty long, so breaking the list up allows me to define each practice and include the individual benefit of each Agile method.  This post defines some hot terms right now- continuous integration, multistage continuous integration, and story points.  Enjoy!

Agile Method: Continuous Integration

How frequently have you merged your code with changes from the mainline, only to find that the result doesn’t build, or it builds but it doesn’t work? Monthly? Weekly? Daily? Hourly? Or worse, how often have you made changes that broke the build, requiring you to quickly correct the problem while getting flames from your team members?

A practice that has emerged to address the problems of integration is called Continuous Integration. The basic idea is that if integrating changes together and getting build and test results on a regular basis is a good idea, integrating and getting build and test results on every change is even better.

With Continuous Integration, all work from all teams is integrated into a single codeline as frequently as possible. Every check-in automatically triggers a build and usually a subsequent run of the test suite. This provides instant feedback about problems to all interested parties and helps to keep the code base free of build and test failures. It also reduces the integration headaches just prior to release.

Agile Method: Multi-stage Continuous Integration

Integration is tough enough when you are just integrating your work with the work of other folks in your small team, or the whole effort is being done by a small team, but when you are part of a large team there is also something called “Big-Bang” integration. That’s the integration of the work that multiple teams have been working on for long periods of time. In a typical project, this integration is done in a phase toward the end of the project. During integration, many problems are discovered for the first time which leads to delays and/or changes in scope.

The real question is, what is a good way to structure this integration so that it will scale smoothly as you add more people to the equation? A good starting place is to look around for a pattern to follow. What are some similar situations? I have found that everything your organization needs to do in order to produce the best possible development organization can be entirely derived from the patterns and practices at the individual level. This approach makes it much easier to understand and much more likely that it will be successfully followed.

As individuals we work in transient isolation to reduce the impact of work in progress on each other. Organizations isolate WIP by using only official versions of 3pty sources and by producing official releases for customers.

Multi-stage continuous integration (MSCI) scales CI to large distributed environments by isolating work in progress at the team level. Changes move from individual to team to mainline as fast as CI allows, but stop on failure. MSCI is particularly important in a distributed environment where fixes to problems exposed by CI can be delayed by a full day

Agile Method: Using Story Points For Estimation, Instead of Units of Time

In my experience, the best unit to use for estimates is story points. Two different people with two different skill sets or levels of ability in an area may take different amounts of time to perform a particular task. Estimating in hours mixes together the scope of the work that needs to be done with the speed at which a particular individual can do that work.

On the other hand, story points are a relative measure of the scope of a user story. Story points separates out the “what” from the “who.” For instance, if you have one individual that is stronger with .Net than with Java, they will estimate a Java story as taking more hours than somebody that is stronger with Java. But they will probably both agree that something that is twice as easy to implement will take half as long to do.

To use story points, you need to create a relative scale of scope. A simple approach is to find a simple and straightforward story that you use to represent a single story point. Then think of stories that are 2, 3, 5, and 8 times larger in scope. You should have a couple of examples for each story point value to take into account that some stories have more test than coding, more documentation than test, etc.

Story points are primarily used for planning, not for implementation. Story points are used to help determine the contents of release by calculating a velocity.

Next up: Backlog, velocity, planning poker and burnup charts.

This past year I’ve attended several Agile conferences, presented at many of our own conferences, and traveled to Agile tradeshows sponsored by some influential industry-leading names. What surprises me most is the variance I see on the answer to this question: How do I do Agile with remote teams?

Some of the pure “Agileistas” will may answer this question in a manner that isn’t very possible for some of us in the real world, with “Don’t do it” or, “Reorg your company.”

I don’t’ know what those people expect here- is it possible that you can convince your management organization to tear down its office walls, move entire teams from across the country into one office space, just because you heard it at a conference that it was going to be really hard to do Agile remotely?

I certainly don’t believe that doing Agile with remote teams is a bad practice, nor do I believe that it’s impossible. Challenging? Yes it is. Easy to mess up? You bet. But there are some simple things you can do to avoid some of the pitfalls of remote organizations.

Agile with Remote Teams

Use face to face communication methods:

I just got the iPhone 4. It has face to face video chat. I also use Google Talk, and this also has video chat built in. It works great! With all the communication technologies we have now a days, there is no reason to avoid personal contact with remote teams.

If the remote team is a faceless organization, it will become a perceived impediment for the local team if there are problems. They wont’ be treated like part of the team, but more like an outside entity that drops code in and risks messing up the release.  We can bring these teams closer together to encourage communication, and allow them to adapt and respond to each other as issues arrive. Creating a persona and human link turns those faceless “code drops” into real people, people who you can reach out to. This gives the team the power to self manage your priorities, impediments and conflicts.

Create Agile ambassadors

We can even take face-to-face chat on the internet up a level. Sending ambassadors back and forth from the remote teams to home base and vice versa creates a human link that is deeper than any piece of technology can provide.  The ambassador’s job is to strengthen this link, because if the link is strong, each side will be more inclined to help each other.

Sometimes having a planning session with the remote team doesn’t give them the overall sense of how important the stories you’re working on might be. They may not feel as if it’s important, and that’s because they don’t know all the juicy details that led up to the creation of that story. Having an ambassador at that site gives that team visibility into all of the bits of information that make one user story important. In other words, the ambassador gives the entire back-story to an iteration (IE the gossip) so they can get a sense of how important something is, it’s not just a priority number in the ITS.

Use Tools That Work Globally

With all of the face-time, ambassadors, and communication, it’s essential that teams have a global view of what’s happening during the development cycle. It wouldn’t make much sense to reach out and then not provide a way to extend that communication on the development level.

Imagine a team where having access to a user story or a piece of code wasn’t easy and available to them? This handcuffs the team tremendously.

Any remote team will need to be able to:

• See each others user stories and tasks
• Enter updates to user stories and tasks
• Diff baselines and branches
• Check out code from remote teams
• Contribute to team discussions and wikis
• Run continuous integration globally

Use Multi Stage Continuous Integration

Using multistage continuous integration lets people take a look at what’s been built, and if it functions correctly, give it to the other team. Having multi-stage set up gives you a way to integrate early and often, but only deliver changes that are “done”.

One of the main problems with remote development is integration, and it’s a double edge sword for most SCM tools. If you isolate the remote team too much, they won’t integrate often. And when they do integrate to mainline, they may break functionality. The problem with this is that they will not be able to respond to that change for 6-12 hours if your team is in another country. This basically means downtime for everyone.

But with multistage CI and AccuRev you can keep that team isolated and integrated at the same time.

Is it possible to do offshore Agile?

I’m not sure if it’s a question if it’s possible, I don’t think we have a choice. Offshore development is a reality that isn’t going away, and the simple answer of bringing teams together to practice Agile isn’t always variable.  Doing Agile with remote teams isn’t’ a choice, it’s a reality.

Continuous integration is about confronting your risk. It’s about giving rapid feedback and responding to problems that you may encounter in your development environment.

If you’re Agile, you know one principle is adaptation. It’s not about sticking to the plan; it’s about exposing and responding to change in your development cycle.  This is why the principles of continuous integration and Agile walk hand in hand.

Delaying the inevitable risk of code integration doesn’t mean the risk will disappear.  Each line of code you change is a risk. Each commit you do is a risk. You can’t avoid the risk by putting it off.  Just like anything, if you were going to bottle up all of that risk, let it ferment for a few months and open it later… I’m pretty sure you wouldn’t want to drink the whole bottle in one gulp. It really only takes one sip to know the concoction is bad.

Continuous Perfection is one of the reasons we bottle up the risk. Let me give you an example.

We set up continuous build on trunk, we encourage our developers to integrate early and often, life is good…. until the day of the broken build.

If it’s broken for even a day, panic ensues, we’ve issued blame to developers for something they should not be responsible for, a stoppage in work for organization. People will feel rushed to resolve issues, the QA team will stop testing, and our release engineer will struggle to roll back code. Essentially, people will be running around like chickens with their heads cut off.

Our management teams do what they do best, they listen to the release team, QA team and the dev team. “Joe Developer broke the build, it’s his fault that we can’t work today”.

When did the broken build become such a big deal?

This scenario may repeat itself, over the course of several weeks our mangers will react, and start asking developers to put in process to avoid breaking the build. Our teams will start to feel that their code must be perfect before they commit. This problem is called Continuous Perfection, it’s the idea that the code must be perfect before it’s committed to the repository.

And doesn’t this seem to work against the principles of continuous integration? The idea here was that we integrate early and often to avoid Integration Hell and the Big Bang merge. We want to confront and work through our problems right now. But now we’ve started to go back to what we do naturally. Isolate changes, put off the inevitable and work off of our private branches. Our developers do local builds to make sure the builds work correctly, and jump through hoops to avoid committing to early in fear of being blamed. We are back where we started, with a huge merge at the end of the project that could fail everything.  We bottled up all our risk, and ended up in a more dangerous place then we intended.

But what can we do to solve this problem ?  To be continued…

By Josh Sherwood, AccuRev engineer

I had the opportunity to co-present a talk on Continuous Integration at the Agile 2008 conference this past week. While there I took the opportunity to attend a variety of sessions. Workshops discussing Business Value, Experience reports describing transitions from Waterfall to Agile, Aspects of Leadership in and Agile Environment and many others. And okay, I’ll admit that I have been a skeptic of a variety of the Agile Processes. Our CTO, Damon Poole, has been an Agile advocate for some time now. He and I have had ongoing discussions about the value of Agile practices. We would talk about the iterative model, going back and forth about the value and how to describe it to different groups. We’d eventually come to an understanding, but there were still many elements that I have been skeptical of and found difficult to see their value.

In addition to the elements of Agile practices that Damon and I would discuss, I started taking a look at other Agile topics. I looked at things like estimation, use of 3×5 cards and even pair programming. I didn’t see the value in the sticky note processes, I mean using a paper process to discuss a digital product? I didn’t see the value in pair programming, especially for consulting companies where your development hours translate directly into billable hours.

But then I came to the conference. I listened to people like Richard Sheridan talk about his experience with Pair Programming, why and how it worked for him. I listened to experience reports from companies like Healthwise, and heard about the difficulties they overcame to improve prior processes and move into Agile. And I spoke with Scrum trainers, who were encouraging their customers and helping them overcome the hurdles of understanding, while remaining flexible about sharing Agile ideas.

What was great about things like the Pair Programming was the team participation. Not only was Richard there talking about their factory model, team members were in the audience, talking about what they take away from the model. They talked about some of the things an individual developer would run into. You know those times, where you are working on a particular problem and just can’t quite figure out how to break the bottleneck. You wander around, get some coffee, or browse the web checking the Olympics. With their pair model, now that bottleneck has two developers working on it. No they’re both not checking their email, they are floating ideas back and forth, bringing to bare the problem and coming to some resolution. With a close proximity to other pairs they are also propogating these ideas farther out amongst the group. Yes there is training and not everybody can manage to work in pairs, but by using this model and actively rotating the pairs they ensure there are no knowledge silos. If someone is out one day there is more than one person who can tackle the problem at hand.

It has been a community of ideas this week. Some people use Scrum but don’t perform retrospectives. Some people model agile practices, but don’t follow a specific practice. Some people are taking the team based processes and developing large scale models for larger development tasks.

As I sit here building out the story for some of our future ideas, I’m encouraged that we can adopt more of the Agile practices than we have in the past, work through some of the confusion those ingrained in waterfall have with new processes, and further the practices that have allowed us to deliver innovation at a faster rate than we could with older models.

Continuous integration (CI) is all the rage these days because merging, building, and testing (shared) configurations early-and-often is a good thing. Actually, it’s a great thing! After all, finding problems sooner rather than later benefits everyone. For some, CI means simply testing compilation. (Phew… it works. Ship it! haha). For those investing time in a full test harness, CI may mean frequently executing a suite of tests at various levels (unit, functional, system) to validate functionality and identify regressions. I’ve even seen other levels of CI to include lab testing, flight testing, or even customer acceptance testing for even the smallest of changes. Regardless of how you ‘do’ CI, I’ll show how I use AccuRev for continuous integration. [Keep in mind that this is one interpretation of the subject matter]

The Pattern. The stream-based nature of AccuRev makes it very natural to define separate areas for development, integration, testing, and release. As seen in my example stream structure, I have an Integration stream as the first point of merging between individual project streams. This Integration stream is a great place to hook up a CI tool [Cruise Control, CC.NET, FinalBuilder, QuickBuild and perform nightly or per-promote builds. I prefer to create a snapshot before doing the build mainly because snapshot creation is atomic and their immutable configurations guarantee reproducibility. After creating the snapshot, I will pull the build from the snapshot name. You could build from the Integration stream directly (similar to the concept of a moving label), but creating snapshots makes it easy to visually identify with the build process and compare good builds from bad builds with simple stream diffs.  [Note: integrating any of the above mentioned CI tools is as simple as telling the build tool to pull code from a stream (by name) and then configure the build tool to execute at some frequency and notify people of the build status]

What about all those snapshots? At first, you may think, “Isn’t this going to create a gazillion snapshots? Won’t that take up a ton of (disk) space and totally clutter the stream browser view?” Well… No.

• Snapshots are cheap. Snapshots are extremely cheap server-side entities consuming ~100bytes regardless of the number of elements they label… so go nuts! Snapshots mark transaction numbers, not elements! I say, always do what you need to solve important problems and answer tough questions even if that means creating a gazillion snapshots; just be sure to organize them.
• Clean up as you go. Your CI build script (build.xml, Makefile, or doBuild.sh) can easily be instructed to remove a snapshot for every snapshot created. I’d recommend keeping around enough snapshots (say 3 to 10) to do valuable work such as comparing builds or serving as temporary baselines for developers who want to reparent. As you can see in the stream structure, AccuRev stores both active and inactive snapshots and it is easy to reactivate any snapshot if necessary (I’ve enabled the stream browser to show both; lower left corner option).
• Group snapshots. I prefer to tuck logically related sets of snapshots behind a locked pass-through stream. The pass-through stream lets me collapse them all as a group and the lock prevents the pass-through stream from being accidentally being reparented.

Tip for very-long build/test cycles. Over the past few years I’ve encountered a few shops with single build/test cycles ranging from hours to days to complete. In this case, the concept of CI is slightly challenging because the notion of frequent builds is constrained. In this case, I’d recommend setting up two distinct test phases; quick and full. The “quick phase” is a quick pass sanity test only performing tasks such as compilation and unit testing — enough to let developers know they can continue on forward progress with little concern. The “full phase” is the full blown cycle, taking hours/days to complete, that completes all levels of testing such as compilation, unit testing, functional testing, system testing, etc. I would execute the quick phase early and often while the full phase may be once per week. As an additional step, I would mark the snapshot used for the full phase with a pass-through stream for the purpose of reporting configuration diffs or letting developers reparent their project streams/workspaces on the latest known good “certified” build.

Interested in continuous integration? Perhaps you’d also be interested in multistage continuous integration…

/happy building/ – dave