Posts Tagged ‘continuous integration’

Continuous Integration & Enabling Local Developer Builds

June 20th, 2011 by AccuRev

Goals of Continuous Integration

If a primary goal of continuous integration is to improve software quality and reduce downstream build and test failures, then perhaps the single most important SCM best practice is to enable automated local developer builds. To this end, developers should be provided with access to a local build environment that mirrors the production build environment, including:

  • Database schema
  • Configuration files
  • Environment variables
  • Shared libraries and files
  • Known compiler/linker/runtime environments

This minimizes the “it compiled fine on my machine” syndrome that plagues many development efforts. Once the build environment is available, every effort should be made to automate the local build, so that each developer can build easily and quickly without having to perform a series of manual steps.

Ideally, the developers should have access to a private build area that uses the same tools and configurations as the production builds. As part of configuring a local build, developers should be trained to take several steps to ensure that local changes will have minimal negative effects once shared with other developers and teams. These steps are as follows:

  • Updating their private workspace to obtain the latest shared configuration code
  • Merging conflicting changes from the shared configuration to their private workspace

By ensuring that developers have known, good build environments, and that they have the latest code with all conflicts resolved, there is high probability that the local build will succeed. If the local build is successful, a continuous integration build that includes the developer’s changes should also succeed.

Shifting to Continuous Integration and Frequently Updating Code

Traditionally, developers tend to put off sharing their changes, sometimes for days, because they don’t want to affect other people too early, or don’t want to get blamed for breaking the build. Unfortunately, this strategy tends to backfire and typically leads to more problems in debugging larger sets of changes after a build breaks. Independent of whether continuous integration is being employed, encouraging developers to build, test, and share their code in small “chunks” is a simple and effective way to improve team collaboration and reduce costly broken builds.

The importance of frequently updating the code in the SCM system is amplified when organizations move towards a continuous integration model. Continuous integration represents a paradigm shift for software development, emphasizing communication between developers about what changes have been made. By breaking down tasks into small chunks that take several hours to complete, developers can “commit” or “promote” their changes frequently and receive immediate feedback about the quality of the changes through the continuous integration build and unit test results. As teams get accustomed to this new approach, developers also gain a sense of progress by seeing not only their own changes build successfully, but also seeing the changes that other developers have made available.

SCM Best Practices and Continuous Integration Go Hand-in-Hand

June 15th, 2011 by AccuRev

There’s no denying that this has certainly been the Agile decade for the software development industry.  It’s evident all around us in this tenth year since the Agile Manifesto was created. Most companies and development organizations today have implemented some form or aspect of Agile methodology into their software development processes. Whether you’re aiming for pure Agile or a mixed/hybrid approach, proven best practices in all phases of the software development lifecycle are crucial to success.

This is especially true in the case of continuous integration, one of the foundational aspects of the Agile methodology. The concept of continuous integration, as defined by Martin Fowler, is “a fully automated and reproducible build, including testing, that runs many times a day.  This allows each developer to integrate daily, thus reducing integration problems.”

With this approach, developers can work more closely in parallel while identify problems and debugging on the fly, accelerating the development process and improving the quality of the finished product.  The benefits of continuous integration are tremendous, but can quickly be eradicated if software configuration management (SCM) best practices are not carefully followed.

There are a handful of SCM best practices that can optimize continuous integration.   Let’s start with a quick look at the first two:

  • Using an SCM system to store and version all source code
  • Utilizing private developer workspaces

Best Practice: Using an SCM System to Store and Version all Source Code

Parallel development and distributed software teams can make tracking changes a daunting task, especially with the frequent changes that occur when using continuous integration methods.

For this reason, it is important to employ a software configuration management (SCM) system to strictly version changes to the code base. In addition to versioning source code, everything needed to build the system should be placed under version control, including the following:

  • Third-party libraries
  • Properties files
  • Database schema
  • Test scripts
  • Install scripts

All developers should have at least read-only access to all files needed for the build and should obtain all such files directly from the SCM system. This approach ensures that developers are working with the latest build environment, and is preferable to the common but error-prone practice of placing such files on a shared file server.

To effectively implement continuous integration, all development groups should work from the same central source code repository so that the latest changes from other developers are easily and immediately available.

Best Practice: Utilizing Private Developer Workspaces

In order to fully realize the benefits of continuous integration, software development organizations need to ensure that developers can remain productive regardless of the overall state and stability of the project source code. To achieve this, private workspaces that give developers full SCM capability should be used. Private workspaces enable developers to

  • work in isolation
  • revert to known “good” states when needed
  • checkpoint their changes
  • share only mature, well-tested code with other team members

The benefits of isolation are bidirectional—it protects developers from incoming changes, and protects the shared code configuration from incomplete or incorrect changes from any one developer. By creating private workspaces, developers receive all the benefits of SCM for their personal use, including the ability to revert to a previous state, viewing and tracking of changes between software configurations, and setting aside changes to begin work on a different task.

Once a new known good state is reached (for example, when a developer completes engineering and testing work on a feature), developers should checkpoint their work, typically by “checking in” or “keeping” the local changes in the SCM system. The checkpoint ensures that the developer’s work is safe on the SCM server and that the checkpoint can be revisited at any time. However, since the changes have not been shared, other developers and teams are not affected.

When a developer breaks isolation and decides to share a code change, he or she is essentially making an assertion that the change has reached a higher level of maturity. This, coupled with the use of local developer builds, helps to ensure that only mature, well-tested code is passed on to the rest of the development team, a primary benefit of continuous integration.

More Agile Methods and Practices Defined

August 30th, 2010 by damonpoole

In the last few posts I have discussed some of, what I consider to be, the most valuable Agile methods for development.  The list is pretty long, so breaking the list up allows me to define each practice and include the individual benefit of each Agile method.  This post defines some hot terms right now- continuous integration, multistage continuous integration, and story points.  Enjoy!

Agile Method: Continuous Integration

How frequently have you merged your code with changes from the mainline, only to find that the result doesn’t build, or it builds but it doesn’t work? Monthly? Weekly? Daily? Hourly? Or worse, how often have you made changes that broke the build, requiring you to quickly correct the problem while getting flames from your team members?

A practice that has emerged to address the problems of integration is called Continuous Integration. The basic idea is that if integrating changes together and getting build and test results on a regular basis is a good idea, integrating and getting build and test results on every change is even better.

With Continuous Integration, all work from all teams is integrated into a single codeline as frequently as possible. Every check-in automatically triggers a build and usually a subsequent run of the test suite. This provides instant feedback about problems to all interested parties and helps to keep the code base free of build and test failures. It also reduces the integration headaches just prior to release.

Agile Method: Multi-stage Continuous Integration

Integration is tough enough when you are just integrating your work with the work of other folks in your small team, or the whole effort is being done by a small team, but when you are part of a large team there is also something called “Big-Bang” integration. That’s the integration of the work that multiple teams have been working on for long periods of time. In a typical project, this integration is done in a phase toward the end of the project. During integration, many problems are discovered for the first time which leads to delays and/or changes in scope.

The real question is, what is a good way to structure this integration so that it will scale smoothly as you add more people to the equation? A good starting place is to look around for a pattern to follow. What are some similar situations? I have found that everything your organization needs to do in order to produce the best possible development organization can be entirely derived from the patterns and practices at the individual level. This approach makes it much easier to understand and much more likely that it will be successfully followed.

As individuals we work in transient isolation to reduce the impact of work in progress on each other. Organizations isolate WIP by using only official versions of 3pty sources and by producing official releases for customers.

Multi-stage continuous integration (MSCI) scales CI to large distributed environments by isolating work in progress at the team level. Changes move from individual to team to mainline as fast as CI allows, but stop on failure. MSCI is particularly important in a distributed environment where fixes to problems exposed by CI can be delayed by a full day

Agile Method: Using Story Points For Estimation, Instead of Units of Time

In my experience, the best unit to use for estimates is story points. Two different people with two different skill sets or levels of ability in an area may take different amounts of time to perform a particular task. Estimating in hours mixes together the scope of the work that needs to be done with the speed at which a particular individual can do that work.

On the other hand, story points are a relative measure of the scope of a user story. Story points separates out the “what” from the “who.” For instance, if you have one individual that is stronger with .Net than with Java, they will estimate a Java story as taking more hours than somebody that is stronger with Java. But they will probably both agree that something that is twice as easy to implement will take half as long to do.

To use story points, you need to create a relative scale of scope. A simple approach is to find a simple and straightforward story that you use to represent a single story point. Then think of stories that are 2, 3, 5, and 8 times larger in scope. You should have a couple of examples for each story point value to take into account that some stories have more test than coding, more documentation than test, etc.

Story points are primarily used for planning, not for implementation. Story points are used to help determine the contents of release by calculating a velocity.

Next up: Backlog, velocity, planning poker and burnup charts.