Archive for June, 2011

Agile Tools: Ten Best Practices

June 29th, 2011 by AccuRev

OK, you’ve decided to transition to Agile development. You’ve done the training, retained an Agile coach, even switched to a new SCM system and acquired your first Agile project management tools. Now what?

Are you worried that you’ve retained some non-Agile baggage and may not be using your Agile tools to their best advantage? No need to worry – here are ten tried-and-true best practices for Agile tool users:

1. Evaluate the Agility of your tool stack

It only makes sense that to use Best Practices for Agile tools, you have to have Agile tools. A good first step is to see if your tool stack fits in with the Agile process. What is an Agile tool ? Simple – it’s any tool that can support one or more Agile practices and fits into the basic Agile framework. That’s it! For instance, a tool which allows you to maintain a ranking of all of the issues that you care about is one that supports the Agile practice of maintaining a backlog. Agile tools need a high ratio of value to effort in order to fit into the short iterations of an Agile project.

For each of the tools in your stack consider the following questions:

  • How long ago did we implement this tool?
  • Do we have the most recent version of this tool?
  • Does the vendor of this tool use Agile development to produce this tool?
  • What are the specific Agile practices that this tool supports?
  • What are the specific features supporting those Agile practices?
  • Are there other tools are better suited to those Agile practices?
  • Are there Agile tools which remove or reduce the need for this tool?

2. Measure value-produced instead of progress-against plan

Rethink how you measure progress with your projects. Toss the idea that progress is measured in milestones right out the window and instead think about value produced. Instead of having unfinished work at the end of an iteration, it’s much better to complete work on a regular basis, get some value going, and have a couple of items which are not done at the end of the iteration that get deferred to the next one.

3. Maintain a master list of all work items

All work needs to be represented in the backlog for it to be really useful. Ideally, all work performed should be in the form of user stories and defects and managed through the backlog so all user stories and defects are available in your Agile Project Management system. The benefit is that nothing falls through the cracks and all work can be managed, tracked, and measured with the same process.

4. Use change packages to link user stories and defects to source file changes

A change package is what you call a complete patch of all changes required to implement a particular change. It’s a rollup of all of the changes that have been applied to resolve an issue including a complete audit trail of who made the changes, when they made the changes, and why they made the changes. Each change package is tied to an issue in the Issue Tracking System (ITS).

5. Fail fast: break up monolithic builds and test suites

I think the best way to be successful is to be able to fail fast. In other words, build the pieces of the software which have changed first, and run the tests which are most likely to detect systemic failures first. This way, with multiple teams working on their own builds, you can get rid of the clunkers quickly and concentrate on those ideas that show the most promise.

6. Use branches to smooth out end-of-iteration activities

Branching enables you to start the next iteration without disturbing the previous one. If QA finds any problems, development can fix them on the previous iteration branch. At some point that branch will be declared “done.” Those changes are then merged into the current iteration. The advantage here is that development did not have to stop, and you still have a stable baseline.

7. Use multiple tracks of development to work toward short iterations

One way to make an easier transition to Agile tools is to dump development tasks into one of two buckets: those that easily break down into small user stories, and those that don’t. The main track is work that can be completed within the iteration, including all testing. The secondary track is for all other tasks. Once a secondary task has been completed, it gets merged into the primary track at the start of the next iteration.

8. Use permissive file access policies to support collective code ownership

There should be as few barriers as possible to making file changes when using Agile tools, so use optimistic file locking – or no file locking at all – and set all file permissions to writeable to support collective code ownership.

9. Base iteration reviews on SCM baselines

Make sure that work shown during an iteration review has been integrated into the mainline and passes all unit tests and other end-of-iteration criteria according to the principles of Continuous Integration by requiring the team to prove its demo work is a single set of binaries built from the same baseline. That baseline must be from the mainline and built on an official build machine.

10. Mirror your Agile workflow with branches or streams

Mirror your Agile workflow with branches or streams so you can get many of the same benefits that tracking state gives you in your SCM change management system.

So there you have it — these ten Best Practices will increase your chances of becoming an Agile rockstar and increase the quality of software you create with your Agile software process.

Best Practices to Optimize Continuous Integration

June 28th, 2011 by AccuRev

There are a handful of SCM best practices that can optimize continuous integration.  This post will look at:

  • Establishing a staging and isolation hierarchy
  • Automating builds at all stages in the hierarchy

Establishing a staging and isolation hierarchy for optimizing Continuous Integration

Proponents of continuous integration commonly suggest branching as little as possible and having developers work directly from the mainline as much as possible. However, this approach has several difficulties:

  • It puts the stability of the mainline at risk
  • It presupposes that traditional legacy branches are the only available isolation mechanism
  • It decreases the flexibility and agility required for fast iterative development

With modern SCM systems, a better approach is to implement a staging and isolation hierarchy for the development process. A staging and isolation hierarchy uses objects in the SCM system to represent the dependencies between development groups and process steps. For example, you may wish to model the following teams and activities:

  • Release engineering
  • Quality assurance
  • Product engineering
  • Component engineering

Each team or activity is assigned the equivalent of a private workspace (variously called “streams” or “branches” depending on the SCM system). Each team then receives the same benefits of private workspaces that individual developers receive.  With a staging hierarchy, changes move from less stable configurations to more stable as they are tested and deemed “good” for the next level. This allows the code to be stabilized as it gets ready for release without developer downtime. It also allows additional separation for each team if needed, so that the team’s changes can be integrated and tested before the components are integrated together.

Topaz Post 3 Best Practices to Optimize Continuous Integration

In this figure, there are four development teams as well as an area for accepting third-party code drops.  The teams are located in different geographical areas. The hierarchy represents the normal flow of changes through development from stage to stage. In the example of the above figure, changes provided by the GUI product engineering team in India flow from individual developer workspaces (not shown for brevity) to the GUI stage, where they can be continuously integrated and tested. Mature changes then flow to the UI_int stage and on to the QA and Release (Rel) stages, again being subject to continuous integration and testing at each stage. The web development team in Austin picks up well-tested changes from the UI_int stage and uses them as the basis of their development work; when the web changes are mature they can be pushed up the hierarchy and subject to broader testing in the UI_int, QA and Rel stages.

Using a development hierarchy provides more opportunities for check-pointing. Every change introduced into the system is a potential source of failure, and thus a potential checkpoint. If a change proves to be unstable, you can return both the source stage and the destination stage back to a previous checkpoint. By contrast, mainline development only offers you a single opportunity for check-pointing, specifically, the state of the main codeline itself. Unless your development process includes “freezing” the mainline for a long enough period to build, test and otherwise validate, the chances of isolating and check-pointing at an appropriately fine level of code granularity are slim, making any available checkpoints stale and of limited utility.

Automating builds at all stages in the hierarchy

In order to give developers prompt feedback about the changes submitted, the code must be built frequently, ideally several times per day. A continuous integration server such as CruiseControl, CruiseControl.NET or Draco.NET can be employed to automate this process. The continuous integration server periodically polls the SCM system for changes, populates the changes to the build server, initiates the build process, and reports the results of the build and unit tests.  It is important to note here that the continuous integration server utilizes the existing build scripts and build environment to execute the build. For example, if make is used to compile and link components written in C, then the continuous integration server will call the makefile to initiate the build process.  Because the continuous integration system uses the existing build, it is important for development groups to devote time and effort to:

  • Making the build as fast as possible,
  • Building automated unit tests and
  • Including unit tests as part of the build process.

Spending time on these items, even if it involves some rework of the build system to make it more compatible with a continuous integration environment, will improve not only the build process but the overall quality of the software release.

When utilizing continuous integration, it is crucial to communicate the results of the builds to the entire development team. Continuous integration system planners should consider a scalable communications method such as e-mail notification or an internal website to display build results.

Continuous integration servers such as CruiseControl come with built-in web reporting that can be easily customized, so that build results can be displayed on LCD panels in common areas at geographically dispersed locations. In this way, team members can easily see and respond to the build results and reduce the “fix latency” often encountered with nightly or weekly integration build approaches.

Continuous Integration & Enabling Local Developer Builds

June 20th, 2011 by AccuRev

Goals of Continuous Integration

If a primary goal of continuous integration is to improve software quality and reduce downstream build and test failures, then perhaps the single most important SCM best practice is to enable automated local developer builds. To this end, developers should be provided with access to a local build environment that mirrors the production build environment, including:

  • Database schema
  • Configuration files
  • Environment variables
  • Shared libraries and files
  • Known compiler/linker/runtime environments

This minimizes the “it compiled fine on my machine” syndrome that plagues many development efforts. Once the build environment is available, every effort should be made to automate the local build, so that each developer can build easily and quickly without having to perform a series of manual steps.

Ideally, the developers should have access to a private build area that uses the same tools and configurations as the production builds. As part of configuring a local build, developers should be trained to take several steps to ensure that local changes will have minimal negative effects once shared with other developers and teams. These steps are as follows:

  • Updating their private workspace to obtain the latest shared configuration code
  • Merging conflicting changes from the shared configuration to their private workspace

By ensuring that developers have known, good build environments, and that they have the latest code with all conflicts resolved, there is high probability that the local build will succeed. If the local build is successful, a continuous integration build that includes the developer’s changes should also succeed.

Shifting to Continuous Integration and Frequently Updating Code

Traditionally, developers tend to put off sharing their changes, sometimes for days, because they don’t want to affect other people too early, or don’t want to get blamed for breaking the build. Unfortunately, this strategy tends to backfire and typically leads to more problems in debugging larger sets of changes after a build breaks. Independent of whether continuous integration is being employed, encouraging developers to build, test, and share their code in small “chunks” is a simple and effective way to improve team collaboration and reduce costly broken builds.

The importance of frequently updating the code in the SCM system is amplified when organizations move towards a continuous integration model. Continuous integration represents a paradigm shift for software development, emphasizing communication between developers about what changes have been made. By breaking down tasks into small chunks that take several hours to complete, developers can “commit” or “promote” their changes frequently and receive immediate feedback about the quality of the changes through the continuous integration build and unit test results. As teams get accustomed to this new approach, developers also gain a sense of progress by seeing not only their own changes build successfully, but also seeing the changes that other developers have made available.