As you start to scale a software development process it becomes apparent that code and user stories have to be merged more frequently. Sometimes changes may flow from one organization to another. This means that you will need to take code from one team, merge, integrate and test those changes with everyone. Each team needs to be able to work on its own schedule, this means that if multiple teams want to work on different sized iterations they can. It also means they can deliver changes as needed and on a regular basis, independent of other teams.

To do this in a traditional Software Configuration Management system, you’ll face two major problems.

1. You have to merge these code changes daily for them to be of any use.
2. You also still need to keep visibility into what stories are shared between teams, because delivering changes from user stories that are not completed in a sprint would be disastrous.

Typically, configuration teams use a single baseline or “trunk” methodology for situations like this, where all changes from each team are delivered to trunk and pulled from trunk as their iterations complete.

In terms of traditional Software Configuration Management, your teams will have to deal with these branching and merging issues. But there are STILL other problems that can happen when you use this trunk methodology:

1. Delivering 2 weeks worth of changes only causes isolation among teams, because teams are working in such a rapidly paced environment.
2. It’s too difficult to pick out each user story from the codebase as it is completed because no tracking is in place.
3. Figuring out the dependencies of those above mentioned user stories is complex.
4. Identifying what changes came from what branch is impossible.

This “baseline pollution” is not scale-able. There are a few ways to alleviate these issues that break the baseline mold. I recommend using a development hierarchy, and manage the relationship of dependencies between branches. This could also include process steps such as integration, quality assurance and code reviews.  A separate code configuration can be used for each step and user stories could simply be drag and dropped between each team, state or branch instead of a merge.

Doing this will increase code stability. As a completed user story is pushed from one stage to the next, the particular change as well as the system as a whole reaches a higher level of maturity. Many traditional SCM tools do not easily support or surface a development hierarchy. AccuRev supports the creation of a hierarchy, gives visibility into the changes at each stage, and enables straightforward merging between stages.

Branching and merging is one of the most critical things a development team must work on over the course of a software release cycle. But there’s a funny thing about branching and merging – it’s usually not thought of as part of the development process. How often do you see a user story called “as a developer I want to merge code to trunk”?

The fact that we often don’t follow a process for the branching and merging of code leads to disarray and pain.  It really shouldn’t be that hard! Teams end up in “merge hell” and deliver changes late to schedule. This problem stems from the way branches are created. They’re often not part of the process and are created from a specific business need, not a from a development practice.

When development teams do create a branching pattern, it’s usually drawn out on a white board or in a Visio document, and is used as a model for the overall development process. While the intention is good, many times these plans become quite complex for reasons that can’t be foreseen.

So, why even create branches, if they’re too complex, and unaccounted for?

Create branches the right way, and use them as:

• Development Branch – A branch created for a development code configurations and builds
• Integration Branch – A special branch for parallel teams to integrate code
• QA Branch – QA branches for QA teams to create builds and environments
• BETA – A preproduction branch for customer sign-off, etc…
• Production – All of the content that ends up in prod

If we take a more philosophical view of what branches represent, beyond business needs, they actually serve as workflows for different aspects of the software development process.

The goals of a branching pattern should always to be to manage a software team’s development process and to make this process as easy and straight-forward as possible. With this process, the team should be able to complete all of the things that need to happen in order to release a piece of software. So how you establish this pattern?

Make Merging Changes Easy and Straight-forward

The funny thing about branching patterns is that we often attempt to create a process, but have a hard time following it. Typically someone from the team will draw the branching pattern on the whiteboard, and even go as far to take a picture of it with a phone to send around to teams.

Unfortunately this process doesn’t scale well when development teams have more than a few people. Change management on the whiteboard isn’t an effective system for ensuring the success of a software release.

Branching patterns should be able to support the development process, and mirror the natural flow of the development team as they work towards a release.

To achieve this, we might want to think of branches as a process management tool, not just a place to put a specific release, patch or development build.  Promotional branching patterns allow for different “states” of code, which is hugely powerful when working through the development cycle. This is a huge topic that is part of larger conversation. To find out more about promotional branching and merging patterns check “A Guide to Branching and Merging Patterns.”

Provide Private Areas for Teams to Check-in and Integrate

Committing early and often is an SCM best practice. Over the years, developers have been told “If it’s not in source control, it never happened.” To avoid this philosophy, teams may require people to check code in everyday so work isn’t lost.

In a practical branching pattern, teams and developers create both private workspaces and branches, allowing them to create builds, releases, and tests of code before they push those changes to other team members. Avoid using mainline type branching patterns that don’t provide code stages; this leads to broken code and unfinished work making its way into a release.

Manage Distributed Teams

Collaborating and sharing code with distributed teams is more complex than ever – teams routinely develop in one location and test or perform other tasks in another location. This distribution of teams strains the development process, yielding security, auditing, and integration problems.

Development teams must appear to be co-located while utilizing the same process with lower complexity. This means code integrations with other projects should happen in real time, so teams can give each other feedback immediately. The branching pattern and process must be followed by everyone, to ensure they are all on the same page.

Understand What has been Delivered

User Stories, bugs, and requirements drive any process, but the ability to see what changes match such items and the location of those code changes are often overlooked in the branching structure.

There is a magical feature that many SCM tools have, called “change-sets” or “Change Packages” in AccuRev. Change Packages associate your changes in the branch with an issue, so you can stay organized and move issues from branch to branch, without having to remember what file went with what issues.

Take these pieces of advice, and they will help if you want to build a release with “finished” issues or if you want traceability into what people are currently working on for a specific release.

It’s here! AccuRev today announced Kando, the seamless integration of Git with the AccuRev server. Everyone at AccuRev is incredibly excited about it.

As many people know (and as we discussed here last week), Git is increasing in popularity among developers working in small groups or collaborating on open source projects. It’s also fast, flexible, and full of developer-friendly features. But when it comes to using Git in an enterprise, the size and complexity of these environments can make it difficult to secure and manage the software development process.

What makes Kando different from other Git integrations?

Take a look at the diagram below. With Kando, Git developers push and pull from real Git repositories. Kando takes all changes pushed to these repositories and replicates them on the AccuRev server. Furthermore, any changes made in AccuRev streams that are mapped to Git repositories are replicated in their respective repositories. This means Git users can just do a pull to get those changes, which allows Git users to continue using Git as usual without interacting with AccuRev, if they choose not to.

 How does Kando benefit Git development environments? 

Kando enables the flexibility of Git and the security of AccuRev by providing:

• Support for enterprise authentication via LDAP and Microsoft Active Directory
• Fully integrated issue tracking system and Software Change and Configuration Management (SCCM) through change-based development with AccuRev Change Packages
• User and group-based access control security measures
• Visualization of development processes using Git through the AccuRev StreamBrowser
• Seamless integration of Git into an AccuRev environment

Take a look at how it works:

To read more about Kando, watch the demo video, and learn about beta availability, check out the Kando page here!

The secret’s out – AccuRev is releasing a seamless security and compliance related solution for the Git community called Kando on January 31st. To get a first look at Kando, register here for the webinar on 1/31/2012, at 1:00 PM EST.

You might be asking yourself, “Why in the world would a company focused on providing software development tools to enterprise organizations with mission-critical software development environments produce a solution for an open source version control tool?” I’ll tell you!

Git is increasing in popularity among developers working in small groups or collaborating on open source projects. It’s fast, flexible, and full of developer-friendly features. Git is a great tool for these smaller and more social types of development projects, and based on discussions about Git with customers, prospects, and analysts, it’s clear that there are more cases of enterprise organizations trying to use Git.

But poke around a few blogs, or read a few articles that discuss the use of Git in an enterprise environment, and I’m sorry, but you will uncover a few issues. As one article in BCW discussed, “Git is a version control system with an attitude of collaboration and sharing. There is practically no way you can enforce a specific pattern of access and sharing. If the people who’re using Git don’t want to follow your rules, the tool is not going to help you much.” Let’s be realistic – Linus didn’t originally design Git for use in an enterprise environment!

So, in which cases do enterprise organizations actually use Git?

1. Android Development

If you want to make changes to Android, you’re going to need Git. It’s unavoidable. This means any company creating mobile devices running on Android and working with Android source files has a real business need to use Git.

2. Linux Development

Same as with Android, if your company has a need to make changes to the the Linux kernel, you are going to need Git. Even if you don’t use Git when making those changes, you’ll eventually have to get them into Git.

3. Working with 3rd Party Vendors or Outsourced Teams Using Git

Similar to the Android and Linux situation, if you’re working with 3rd party vendors or outsourced teams who require that you merge your changes into their Git repository, you may be forced to use scripts or bridges to get your changes from your SCM into Git or vice versa, and that’s not a small task.

4. All of Your Developers Love Git

Let’s face it – Git has a cult-like following in the development community. Developers love Git because it’s fast, distributed, flexible, fairly easy to learn, and has a ton of developer-friendly features. It’s developed by developers for developers. Even if you understand the issues Git has with scaling in enterprise environments, it’s difficult to avoid Git when lots of your developers are pushing you to switch.

I was browsing reddit.com the other day and ran into this post:

Yup. It’s true. The tried and true development approach of Waterfall that we’ve been using for years was an example of what NOT to do for software development.

From the Wikipedia article: The first formal description of the waterfall model is often cited as a 1970 article by Winston W. Royce,[3] though Royce did not use the term “waterfall” in this article. Royce presented this model as an example of a flawed, non-working model (Royce 1970). This, in fact, is how the term is generally used in writing about software development—to describe a critical view of a commonly used software practice.

That’s what’s amazing about waterfall, and the agile transformations that seem to be taking the industry by storm. Maybe we all know deep down there is a better way to develop software.

I hope someday we don’t look back on Agile the same way we look back on Waterfall. I don’t think it will happen for the simple reason that Agile doesn’t have one methodology tied to it. Agile can mean a simple set of practices to help with software development, but it’s more like a mission statement as opposed to a plan.

SCM tools have a profound effect on the day to day life of a developer. These types of systems have either helped or hindered development teams deliver software. SCM systems are like the “hub” of a development team. It’s where teams artifact important work, integrate changes, save important ideas and add features for customers. It’s the center of our development universe!

It’s all about the developers. They need to be free to innovate and get changes out the door quickly. But they can’t if they are stifled by tools that get in the way. Tools need to be able to ENHANCE the software development process. Many people think that source control is just a place to checkin / checkout code. But it’s more than that, it’s where the software development process comes to life. If the SCM system isn’t up to the task of a complex development process, developers can’t innovate.

Sometimes it’s hard to understand that you have a tooling problem, even if it’s staring you in the face. Think of an old pair of trusty sneakers that you have at your house. We all have a pair, they are many years old, beat-up, dirty, torn… but we still wear them. Our feet hurt when we wear them, but for some reason we refuse to get rid of these old sneakers. Until one day (usually after a sprained toe) we decide to buy a brand new pair and after a little breaking in… WOW our feet feel great! Why did I keep the other pair so long?

Software tools are often like this, there is an “if it ain’t broke (too much), don’t fix it” attitude. We often keep tools too long after their expiration date. You’ll hear it from your development team, moaning about the pains of merging code, switching workspaces, checking out … it’s enough to make you cringe. But still we don’t change. Your old SCM is the sneaker, and collectively as a group you and your team have a hard time recognizing when your feet hurt.

‘Tis the season to make unsupportable predictions for the future.  Despite my prior record (and I remain surprised that we don’t yet have personal jet packs) I’d still like to share a long-range weather forecast for the software industry.

You’ve been warned.  From here on, you’re on your own.

Prediction 1: Everyone Will Claim They Are Agile

And 50% of them will be wrong, just based on the Nokia test.  And of the rest, half won’t get any value from it.

There are a lot, and here I really need to underline a lot, of bad development practices out there.  For every organization that is killing it with Agile, there are five (my agilesta friends say ten) organizations that are limping along, delivering buggy code to their customers, late, and missing committed functionality.  And often all three.

This “Going Agile Without Knowing How” problem is probably an inevitable result of the success the early-adopter teams had with Agile methods.  For instance, when I watch The Olympics, figure skaters make skating look effortless.  When I do it, I look like a drunken hippo and hurt my butt.  It’s hard to stop and remember that these athletes, in addition to good genetics, spent years at the rink with their coaches learning, trying, failing, and improving, before they got in front of the TV cameras.

Agile has crossed the chasm, and the great majority of organizations have too few people, with too little coaching, and hardly any tooling.  Sure, your boss doesn’t realize how useless your stand-up meetings are, or that your code isn’t fully tested at the end of a sprint, but she’ll eventually see that your customers are not happy.

Prediction 2: Development for Mobile Devices Will Still be Small

Yes, Mobile is really big, and moving fast.  It’s just that the great majority of the work to support useful mobile apps remains in the back office.  When we’re finished inventing new ways to swipe our coffee-stained fingers across our screens, the value of the great majority of our apps is back in the glass house, running Java and C++ on big ‘ole honking (virtualized) servers.

The development problem in the era of proliferating small platforms, remains the problem of dealing with large, complex, data and it’s interactions.

Sleep well, Larry Ellison.

Prediction 3: The Gap Between Pros And Amateurs Will Grow

Every new software technology “spike” rewards the early adopters with higher productivity, which can level the playing field for the “newbie” or occasional software developer.  But as application complexity grows, as the platforms become more complex and development environments become richer, the professional advantage becomes more significant.

There isn’t much of a disadvantage in time-to-market for the young developer, maybe working on his laptop with open source tools and no identifiable process. The difference between them and a team of experienced professionals, working with industrial strength tools and procedures, and building apps that run businesses on virtualized hardware in a web connected world, is in “value created.”

Created Value is a concept that we’ve all been learning during the past ten years of Agile evangelism.  Created Value is measured at the customer side, and primarily by the classic metrics of software: how does the software help me get my job done better and faster?

There was a time when the lowest cost labour source for a software project was the key criteria.  Over the past two years, software projects have been revisiting their decisions as they’ve seen the crippling effects of buggy, unmaintainable, badly architected products.  We’ve seen the evidence with a hiring boom in the US for developers and QA alike.

In short, in 2012, we’ll see a renewed focus on quality of development, over quantity.  And a better appreciation for the talent, tools, and techniques, that create it.

I hope I don’t end up with a seized engine on the side of the road, but if I do, I’ll know I should have had that oil change. I hope I don’t end up on the Worst Dressed List, but if I do, at least I’ll know I should have given away those old shirts.  I feel sorry for those on the “Worst Agile Implementation” list who don’t even know they’re there.

In the past few months I’ve had the privilege of talking to approximately fifty organizations about their Agile implementation.  Most of them are doing well, and many of them have great insights about how they customized Agile to fit their process requirements.  But some of them really Say the Darndest Things.

“We do Scrum, it’s just the rest of the company doesn’t.”

“So first we break the requirements specification into pieces and call each of the pieces a story.  Then we do our iterations and pass them off to the release team.  We’d sure like to get Product Management, QA, and the customer involved, but they don’t want to.”

There are a lot of places an Agile approach can add value, and I’d hate to adopt a “waterfall approach to going Agile”, but you’re really not doing Scrum.  The biggest chunks of value, the incremental use of customer feedback, and going from “completed state” to “completed state” in each iteration are lost if you can’t get more support.

“We’re Agile until the development is done.”

More than once I’ve been speaking with an earnest development leader who’s describing the Scrum process.  They’ll launch in, with obvious pride, and tell me how they’ve gone to two week iterations, do standup meetings, and work from a backlog.  “Terrific!  And how do you do QA?”

Oh, yes, of course they do QA, silly!  In fact, they demo the completed development to the QA team every sprint review and send it off to get tested.  Sometimes, unfortunately, QA actually finds some bugs that need fixing.  So that’s why they put the sprint on hold for a while to fix the bugs and loop them back into QA “’cause we don’t want to wait an entire sprint before they can restart the testing.”

The other side of this one is the guys that take the old “Release Tail” loophole for all it’s worth.  “Yes, Lorne, we’ve been agile for three years now.  We do Scrum, unit testing, standups, and play in the World Series of ‘Planning Poker’.  We do that for about six weeks, or until the release.  Then we have a three month release testing tail, which follows a ‘modified Scrum process’ … the project leader estimates the amount of work on each bug QA finds, and assigns it to a developer.  Sure, sometimes we have to work on new functionality during the “release testing tail” … you can’t expect the customer to stop needing improvements for three months!”

Folks, I don’t think I’m sharing any great trade secret when I tell you the QA process needs to be completed before the story is considered “done.”  I don’t want to be Klaus Fuchs of Scrum, but here’s the secret: you’re going to have to invest more in testing up front.

“We do continuous integration every night.”

I blame the education system: how’s an engineer supposed to know what “Continuous” means when we have “social promotion!”  Now some people understand the idea of continuous integration, and made a conscious effort to make it more “Discrete”.  Some companies I talked to had broken builds that lasted for a week.  You’d rather have a child repeating “Mummy” every 30^th of a second before you’d like to get an email every five minutes saying the “Build Failed.”  I get it.  And if the email was going to your boss too, well, you don’t have to be Dogbert to know that’s a bad idea.

Builds are going to fail.  Get used to it.  The problem is not that the build failed, but that you couldn’t fix it.  Good practices are to have the team drop what they’re doing when the build fails and hop on fixing it.  If they can’t fix it, it needs to get escalated *pronto*.  Better is to have the team do local builds and unit testing before they check in.  Best Practices are to divide up the build process by team and stage of development, so your team only pollutes itself, not the rest of the development org.

“We don’t need training since we can use the internet.”

Uh huh. So I guess the schools will be shutting down any day now.  Not that the Internet might not turn out to be a useful aid someday, but the software development process is a hands-on activity.  And similar to other hands on activities, like dancing or carpentry, you can’t learn to do it by reading a book.  You’re going to need to get some experience with the process before you understand how to run a sprint review or a stand up, how to estimate stories, and how to work with your QA partner.

Now if you’re a hobbyist and working for free, your time is cheap, and there’s no reason not to use trial and error as a learning method.  But if you’re getting paid, and your work is important, you really don’t want to waste four sprints figuring out what someone can help you get right in sprint two.

I’m hoping my surgeon, pilot, and barber got a few lessons before it was my turn.

Finally…

No one has to pass a test to call themselves “Agile,” nor should they. Agilistas don’t have a monopoly on the right way to develop software.  But when people believe they’ve made it to Agile without using critical Agile concepts like time boxing development or getting to “done”, they’re missing the real value.

Did you know that CM systems back in the day were basically people? This is where the term “check-in” & “check-out” comes from- it refers to the days when there where actual software librarians would record peoples changes and check them in and out like books on disk or punch cards. It’s mind boggling to think of software this way.

If I was to ask software developers today what “software configuration management” was, they would probably say “SCM? Like Subversion?” Incorrect! You need to trust me on this one, SCM is not the same as a version control system. Yes, your version control system is an SCM tool (confusing?) but SCM is a broader discipline and technique that encompasses the management of change in software.

The introduction to the IEEE “Standard for Software Configuration begins with:

SCM constitutes good engineering practice for all software projects, whether phased development, rapid prototyping, or ongoing maintenance. It enhances the reliability and quality of software by:

• Providing a structure for identifying and controlling documentation, code, interfaces, and databases to support all life cycle phases
• Supporting a chosen development/maintenance methodology that fits the requirements, standards, policies, organization, and management philosophy
• Producing management and product information concerning the status of baselines, change control, tests, releases, audits, etc.

Let’s be clear- all of the things on this list do not fit under the heading of your version control system. Many of them will require practices and policies to maximize your development efforts and methodologies. With version control, release engineers will still have to perform some of these SCM related functions:

• Merge early and often
• Enforce a workflow for development teams to follow
• Record and have full visibility into all of the changes that were made
• Write build and compiler scripts
• Automate builds, deploys and tests
• Understand the dependencies between projects and code
• Maintain the development environment for a team
• Be responsible for the final product going out the door

That’s just the tip of the iceberg. A talented release engineer or SCM expert can do all of those things independently, but his or her job would be a lot easier with SCM tools that can automate and facilitate the necessary practices and processes. (This includes version control, compilers, debuggers, editors, continuous integration machines, automated deploy, and the ITS system.)

At it’s core, SCM answers the question “Somebody did something, how can one reproduce it?” In addition it’s about understanding and establishing relationships among items that are likely to change. It’s a tricky job, not one that’s easily understood. We have to understand the relationships between versioned artifacts, like code, hardware, documents, design models and even directory structures. In addition we have to do all of the necessary things to make those versions valuable to our organization. We have to design process, workflow, automation, build automation, reports and security.

With all of this, don’t tell me that SCM is the same as version control. Trust me on this one!