Branching and merging is one of the most critical things a development team must work on over the course of a software release cycle. But there’s a funny thing about branching and merging – it’s usually not thought of as part of the development process. How often do you see a user story called “as a developer I want to merge code to trunk”?

The fact that we often don’t follow a process for the branching and merging of code leads to disarray and pain.  It really shouldn’t be that hard! Teams end up in “merge hell” and deliver changes late to schedule. This problem stems from the way branches are created. They’re often not part of the process and are created from a specific business need, not a from a development practice.

When development teams do create a branching pattern, it’s usually drawn out on a white board or in a Visio document, and is used as a model for the overall development process. While the intention is good, many times these plans become quite complex for reasons that can’t be foreseen.

So, why even create branches, if they’re too complex, and unaccounted for?

Create branches the right way, and use them as:

• Development Branch – A branch created for a development code configurations and builds
• Integration Branch – A special branch for parallel teams to integrate code
• QA Branch – QA branches for QA teams to create builds and environments
• BETA – A preproduction branch for customer sign-off, etc…
• Production – All of the content that ends up in prod

If we take a more philosophical view of what branches represent, beyond business needs, they actually serve as workflows for different aspects of the software development process.

Branching and merging is one of those things in software development that tends to drive most developers right around the bend. It’s understandable why so many fear branching and merging – after all, so many things can go wrong once you veer off the baseline and wander into uncharted territory and try to get back. But instead of fearing the beast, here are two tips to help you drag it up out of the basement, smack it around, and show it who’s boss.

Just so we’re all on the same page, branching and merging is defined as the process of duplicating part of a software development project baseline so that some parallel development can take place, say, to fulfill a customer’s change request, then once completed it’s merged back into the baseline.

The fat can hit the fire in two places: first, determining when and why a branch should be started, and then when and how it’s merged back into the main development baseline. Done poorly, both of these actions can result in confusion, errors, and delays. No wonder branching and merging gives developers night-sweats.

Because branching and merging has the potential to be problematic, some developers unintentionally make it worse by either branching too often or merging too late. The result can be increased risk in errors and/or a decrease in productivity.

So what’s a developer to do?

Start by determining what should rightfully constitute a branch – does a branch get triggered by a change request? How about a specific integration effort? Perhaps it’s based on system architecture? These are just a few of the common branching strategies, so you need to decide which one provides you with an acceptable balance of risk versus productivity. Do you want to branch and merge early and often, reducing risk but also slowing productivity? Or are you willing to roll the dice that things go smoothly during development so you’ll hold off on merging to potentially increase productivity and value?

Where things can get really sticky is in the merging process.

So my second tip is to make sure you can fully track all your file changes, especially across code lines, or you run the risk of some seriously broken builds and test failures. At AccuRev we have a branching model that we call streams which provides some great code flexibility to make branching and merging as fast and efficient as possible. Stream branching easily accepts parent code while allowing you to then push code out to other streams. All file versions are stream-based, making merging easier because the entire file change history is available and automatically tracked. Pretty cool.

Here’s the big pay-off – because branching and merging is made a lot easier with streams, development teams can branch more often and more confidently, reducing risk without seriously impacting productivity. Perhaps best of all, it helps you avoid a potentially-disastrous end-of-release merge dump that could result in costly and time-consuming fixes and release delays.

So let’s recap: avoid a branching and merging nightmare by carefully considering the branching model that best suits your needs and your level of risk versus productivity. Make sure that your methodology provides maximum visibility into the process and enables you to fully track your file change history, enabling better decision-making and ensuring more effective merging that’s less prone to errors. The result? Better product,faster, and with less errors. How sweet is that?

Are you frequently called on to perform the daunting task of ‘the merge’? Does tension mount and do groans grow louder when it’s time for a merge?  Are only a select few are willing to stand up and do the work?

Maybe these stages describe your merge process:

1. Denial: It’s never going to work (in a timely manner).
2. Anger: Why do I have to do the merge?
3. Bargaining: I’ll work on maintenance and support if I don’t have to do the merge.
4. Depression: How many compiler errors? Why won’t the jsp work with firefox?
5. Acceptance: Not really, it’s compiling and (mostly) passing tests.

Before we try to get you through the 5 stages of merge, let’s a take a look at divergence, the cause of a complicated merge.

Converging Divergence

There are lots of reasons for divergence, and as many for merge product lines back together.

Maybe you’re delivering functionality to specific customers, creating customization and personalization of websites, internationalizing your product, even moving from Windows to Linux platforms. So what happens? These areas diverge, developers refactor code, and the product lines look very, very different over time. Sure, you try to propagate changes across the product lines, but each implementation is slightly different. Eventually you realize this is a maintenance nightmare, and some shared areas should be brought back in line (if not the whole product line moving to a single source).

Now comes the hard part. A select few of your developers have to start piecing the code back together. Determining what changes are acceptable, reviewing the functionality, and hopefully getting the appropriate test coverage to verify that the desired changes have been brought back into a single (or at least cleaner) product line. Here’s where tools really shine (or falter).

Now, I consider myself pretty experienced in the merge department. I’ve worked on a number of merges of different sizes, dealt with multiple languages (English/Italian), and even migrated to different platforms (Moving from Windows to Macintosh).  Sometimes, I found, tools were a complete hindrance- I remember back in the day, using PVCS or CVS was every man for himself. You just got it working, never mind worrying about who made what code decision, and hopefully extensive tests identified the problems. More modern tools, AccuRev in particular, give you another dimension with merge that I find to be an essential.

The Merge Master

AccuRev version tracking, and the changes it tracks (content, name, removal) really help in hunting down the straightforward changes that can result in truly subtle changes in product behavior. Simply knowing that files have been removed in one product, and merging those changes over with a click of a button can eliminate behavior that is hard to determine, especially when the developer doing the merge had a reasonable belief that the files may exist.

Even better, large or midsize merges, can be easily reviewed, grouped into areas of the code and dealt with in a manageable manner. I performed a merge that merged internationalization changes into a product at AccuRev. There were over 7000 files that were altered, added or removed that needed to be managed.

Simply grouping the files by directory structure quickly identified which were doc, help, client, java, etc. Then these areas could be dealt with by developers in the specific areas. Further, because AccuRev performs simple merges quickly, I was able to perform the initial merges in these areas without having a strong need to understand them, and leave the complex changes to those who specialized in particular areas. This further reduced the amount of effort (and pain) that other team members experienced.

In this 7000+ file example, only about 400 files actually required more specialized investigation. And of the 400, only about 50 had truly complex changes where both sides of the merge made what looked like contrary decisions. Here is another place where AccuRev shines. By looking at the changes made at different points in each of the product lines, and working with change packages that described what the developers where trying to accomplish, I was able to make intelligent decisions about how the code should integrate, instead of just picking ‘he who changed it last’.

The two product lines in this case had diverged for about 9 months, and had about 10 developers making changes in each product line. Even with the amount of changes that occurred over this time period, it only took a single developer about 2 weeks time to bring the two product lines together. It took only a couple of days to complete the initial code process, walking through the changes and picking (via the AccuRev GUI) which of the changes to take, and editing inline where it was obvious.

During the first week, we were able to finish digging around developer changes via the AccuRev version browser in order to help resolve initial compilation failures. This work brought the product to the point where it was compiling on multiple platforms (which is usually even a problem with nightly builds). After another week of going through the validation and user tests, the merge was stable enough to be considered the baseline for the next release. Of course development had already occurred during the two weeks that the merge process took, but with the version tracking AccuRev performs with merges, it became almost trivial to bring in the new changes- within 1 day they were validated and ready to be the new baseline.

Sounds like acceptance to me.

You spent the last 2 weeks working on 10 user stories for you sprint. Your team has completed 8 of the 10 user stories, and now it’s time to show a working demo for your sprint review. Sounds like a typical scenario right?

Here’s the problem, the 2 remaining user stories have partial work associated with them. That code is in your source control system, and in order to correctly demo or ship what was accomplished in your sprint, you’re going to have to do some work to create a “done” product. You will have to subtractively merge those changes, or put them on another branch, then re-test the application, redeploy.. check the changes again etc. There is still a lot of work to finished in getting this sprint to “done.”

One way to get around this is to associate user stories with code changes in your SCM tool.  AccuRev already creates a link between your user stories and code in the form of change packages. Creating that link between the user stories and your code is the first step. But using a powerful story-driven process out of this linkage is the key to “getting to done” in an Agile environment.

Your user story will move through different stages as you move through your process. One user story may have these states:

DEV- Currently in progress with dev team, or “coding”
QA- Coding finished, ready for QA to test
UAT- QA finished ready for user acceptance testing
PROD- Tested by QA and users, ready to ship

However, while  user stories move through this process, the code doesn’t follow. Traditional SCM systems are designed around single branches and a long and lengthy merge processes. Fast paced Agile environments expose those limitations as development teams struggle to ship code to customers at the end of iterations.

During that iteration, testers may want to test completed user stories, but need a stable configuration to do so. If you’re using a traditional waterfall branching and merging based SCM tool, you may end up with some user stories ready for QA, and some still in the DEV stage, creating a poor testing environment and broken builds. As a result, developers often delay committing code so changes can be implemented, tested, and passed to QA before they integrate.

During your retrospective of the previous iteration, you may decide that parts of your process need modification. For example, you may decide to add a code review process right before testing. Your SCM system has to be flexible to enable this new step without time consuming tasks, like re-writing scripts.

SCM and an Agile Process

The trick is to have an SCM system that can help you manage and enforce an Agile process. Code should be able to follow the same process user stories follow. If a user story is in QA, all of the code needed to test that issue should reside in a QA configuration. The same goes for DEV, UAT, PROD, or any other process that is part of your environment.

A stream structure is an easy way to  process change – streams can be added to adjust your process. In a few clicks, you can add a code review between a DEV, and QA step.

Agile and SCM: Avoiding Agile Merge Hell

As you start to scale Agile, code and user stories have to be merged more often. Sometimes changes my flow from one organization to another. This means that you will need to take code from one team, merge, integrate and test those changes with everyone. Each team needs to be able to work on their own schedule, this means that if multiple teams want to work on different sized iterations they can. In addition they can deliver changes as they need to on a regular basis, independent of the the other teams.

The problem here is that to do this in a traditional SCM system, you would have to merge these code changes daily for them to be of any use. You also still need to keep visibility into what stories are shared between teams, because delivering changes from user stories that are not completed in a sprint would be disastrous.

Typically, the type of configuration people use for this is a single baseline or “trunk” methodology, where all changes from each team are delivered to trunk and pulled from trunk as their iterations are completed.

Working within an Agile context, your teams will have to deal with these branching and merging issues. But there are other problems that can happen when you use this methodology:

1. Delivering 2 weeks worth of changes only causes isolation among teams.
2. It’s too hard to pick out each user story as it completes from our codebase.
3. Figuring out the dependencies of those user stories is complex.
4. Being able to identify what changes came from each branch is impossible.

This “baseline pollution” is not scale-able. You can get around this by using a development hierarchy, and manage the relationship of dependencies between branches. This could also include process steps such as integration, quality assurance, and code reviews.  A separate code configuration can be used for each step and user stories could simply be drag and dropped between each team, state or branch instead of a merge.

Doing this will increase code stability. As a completed user story is pushed from one stage to the next, the particular change, as well as the system as a whole, reaches a higher level of maturity. While many traditional SCM tools do not easily support or surface a development hierarchy, AccuRev supports the creation of a hierarchy, gives visibility into the changes at each stage, and enables straightforward merging between stages.

To borrow from a common print journalism trick, consider this the cleaning out the desk drawers of my AccuRev mind.  Or if you tend to avoid newspaper columnists, especially sports ones, then let’s just call this a random “did you know” or “what’s new” post.  I think the last time I did one of these was in the AccuRev Version 4.5 days, and we now entered the days of AccuRev 4.9, so there might be a few things tossed in here that aren’t necessarily fresh off the presses…

AccuRev 4.9 introduces the long sought after “stale filter”.  Actually, it’s an Update Preview search, but it shows not just (stale) files, but also anything that will change at update time, like moves and new elements.  You can even perform actions in that search pane, like Diffs, prior to updating…

AccuRev 4.9 also brings you the -t <time_spec> argument for both stat and pop.  This lets you provide a transaction or time stamp to the command so you can reference a configuration of code at a point in time without having to utilize a time-based stream.

Sticking with command-line tidbits, you can now use pop -D to populate elements without including their entire directory structure.  Very useful in situations where you might have different applications contained under a root folder yet want to populate them at a top level.

Another really helpful enhancement is to the Deep Overlap search.  Now when you Merge a file that is in Deep Overlap, AccuRev changes the status to (kept)(member) in the filter so that you know which files have been operated on.  You also have the ability to promote from this search window, and subsequently the promoted file will no longer appear in the filter.

If you haven’t started using the AccuRev WebUI, you’re really missing out.  One lesser known capability in the WebUI is the ability to take a regular query, Group by any field, and automatically convert it to a bar, line, or pie chart.  This report can then be sent and accessed as a URL, and obviously can be viewed by users without installing the classic GUI.

To further bind the classic GUI and WebUI together, AccuWork issues in the classic GUI now contain a hyperlink “Issue URL” which takes you directly to the URL rendering of the issue in the WebUI.  And for those with a very extensive stream structure, have you checked out the St\ream Filter to personalize and only display streams that you care about?

For folks running Continuous Integration, or for those who just want to be able to use command-line to see what’s changed in a stream over time – including inherited content! – I think you’ll be happy with this one.  You can use the accurev diff command to compare the contents of a single stream with itself over a period of time, without having to create a time-based stream.

Have you noticed that when using the Stream Issue Mode in the StreamBrowser, you now see Incomplete Issues right in the “Show Active…” box attached to streams?

And lastly for now, bonus points for anyone who can comment here to point out the AccuRev command-line typo that still works as if it is spelled correctly.  AccuRevers or former ones need not apply 

__________________________________________________________________________________________________________________________________________________

I work for a large global enterprise software company with over 500 users, that has been using AccuRev for over 4 years.

We are currently managing a Beta release of our main product line.  The management of the source code is easy thanks to AccuRev, and here’s why:

1) Stream inheritance is just plain awesome!

We manage hundreds of streams to make up our suite.  We are able to promote code up through task streams, to integration streams, to qa streams where we ship.  For the Beta, we have child streams qa.beta, and our build machines use these streams.

2) Snapshots rule!

Snapshots by definition are immutable.  So you know what you built, when you built it.  Our build process creates snapshots automatically, and we fetch code based on snapshot.  We have 100% confidence in knowing what went into a build, and our snapshots are named to a pattern i.e. <parent stream>.yyyymmddhhmmss.build<build number>.  That gives one a lot of info about a build.

3) Timelocks let you stabilize code, even if set the next day!

We built our Beta release on snapshot, and the next day we retroactively set time locks on the qa.beta streams based on build time, and we access locked them down.  Now we can finalize the Beta, and still allow other integration and qa work to happen.

In summary, if we were using a traditional branch/merge SCM, this would be a lot of work and require dedicated personnel.  Instead, we did this part time and the StreamBrowser makes is so easy to see what code is where and why.  There is no confusion with views or other techniques used by other vendors.

Yesterday I was a panelist for a Webinar on agile tools, focusing on software configuration management (SCM), build and software process automation (SPA), the latter term referring to the set of defined, repeatable and measurable automated development workflows that engineers use to transform requirements into shippable software products. Contrary to what I’ve read about the disdain that some agile devotees have for tools, most of the attendees were hungry to know what features their SCM tool should have in order to support agile software development and SPA. Here are some of the highlights, and of course, my take on why I think AccuRev is the best tool for agile software process automation.

There are five key feature areas that an SCM tool needs to support in order to be ready for agile:

* Support for flexible process models

* Continuous integration support

* Support for issue-based development

* Efficient branch and code management

* Private version controlled developer workspaces

Let’s take a look at each of these in turn.

* Support for flexible process models. Agile is often one of several processes being employed within a software development organization. Unless your SCM tool is flexible and process-neutral, you will have a hard time implementing agile (say, for product development) and more traditional processes like waterfall (for example, for product maintenance work) in the same SCM tool. AccuRev streams are a natural way to model any process, and thus are a good fit when agile needs to coexist with other development processes. As for software process automation (SPA), AccuRev streams again are a great fit, since they enable users to model any arbitrary stages of code transformation that a development team sees fit to define as part of their process. By adding triggers and workflow to a stream hierarchy, teams can implement SPA directly in AccuRrev.

* Continuous integration support. Continuous integration is one of the core process elements associated with agile development. By building and testing frequently and acting on the results of tests, teams can uncover defects or test gaps earlier in their development cycle, saving time and money compared to such discoveries late in the cycle. But continuous integration goes beyond just testing the nightly build. With multi-stage continuous integration in AccuRev, code is automatically promoted up the stream hierarchy into more stable configurations as it passes tests. At each stage, continuous integration takes over to build and test, typically with a wider scope of testing as the code nears the release stage. Legacy SCM tools make this type of automated integration factory somewhere between difficult and impossible due to the complexity involved in setting up the hierarchy and in automatically moving and merging code as it flows up the hierarchy.

* Support for issue-based development. Apparently there is a lot of contention about the need for filing issues and defects in agile development. This has puzzled me greatly. While I’m in favor of developers identifying and fixing issues as they are discovered, you lose valuable process information when a defect or enhancement ticket is not filed and later associated with a code change. Without an issue that describes what the problem was, someone looking at the code changes for audit purposes or for group code reviews is at a disadvantage. Why was this code change made? Is it related to other changes? How long did it take? Was it done to fix a bug or to add a feature. In AccuRev, issues either in the integrated AccuWork issue tracking system, or in a 3rd party issue tracking system, can easily be associated with code changes via the AccuRev Change Package mechanism. This establishes basic traceability between issues and the code changes that developers make in order to satsify the requirements of those issues. Issue-based development is well-defined, repeatable and measurable – all hallmarks of good software process automation.

* Efficient branch and code management. Any time you’re working on more than one project, you need to isolate that project’s code from other projects. With agile and multistage continuous integration, even a single project requires multiple code lines in order to separate in-progress code from unit tested code from system tested code that is ready for release. If an SCM tool makes branching, merging and labeling difficult, teams tend to practice branch avoidance, which I sometimes like to call “fear of branching.” This is a classic example of letting a tool dictate what processes you can implement. In AccuRev, streams replace branches as the mechanism for isolating codelines. Since streams are represented inside of AccuRev as data separate from the actual file versions, creating streams is fast – really fast, like, a second or two – and managing a system with hundreds of streams spanning multiple projects and processes is easy.  For continuous integration, AccuRev snapshots and time-based streams are also fast and easy to create and manage, and give users a straight-forward way to “label” an interim or milestone codeline without having to place markers in thousands of source files.

* Private version controlled developer workspaces. Software developers are the heartbeat of any engineering organization. Executives at any development shop will tell you that hiring talented engineers and keeping them well-tooled and productive is the single largest challenge that they face. For agile, this is even more of a challenge, since coding cycles tend to be shorter, and thus anything that gets in the way of individual or team productivity tends to have a greater negative impact on the project. Private version controlled workspaces like the AccuRev workspace model improve productivity, since they enable developers to work in isolation (while they are ‘heads down’ coding). Private workspaces in AccuRev also give developers full SCM capabilites in their workspaces without the need to share in-progress code prematurely. By using the ‘keep’ operation, developers make safe copies of their work in the AccuRev repository, and later can ‘promote’ the code to an integration stream to combine their work with that of their teammates. Individuals are more productive in this way, and if continuous integration builds are frequently testing the integration stream, so are teams.

In a nutshell, agile requires tools, and these tools need to support different modes of operation than with other processes. SCM can help or hurt you in setting up and executing an agile process, so these guidelines are a way to help you get your SCM tool ready for agile - easy of course if your tool is already AccuRev!

If you’re interested, you can view the webinar recording.

Is Your Software Development Environment Agile-Ready?
Free On-Demand Webinar