It’s 1998. I’m just starting my job at Rational Software, supporting ClearCase. During my training and through my first few months handling customer issues, I developed a deep understanding of the complexities of SCM and the heavy-handed centralized control that ClearCase provided its customer base. Wrappers, triggers and script automation were the key to a tightly controlled, centralized development environment.  In the developers’ eyes, dealing with the version control tool was just a part of doing their job.

Developers like to code, not “waste time” with tools or infrastructure. They are creators and want to create their masterpiece. Check-in, Check-out, Merge, etc. are not ways developers want to spend their time, especially if they are forced to deal with poor performance from an SCM tool, whether it’s by the tool itself, or the myriad of wrappers that were forced on top of the SCM tool their company chose. In a developers eyes, any time spent NOT writing code is wasted time.

However, despite the negative ramifications of SCM centralization and control to the developer’s productivity, there are absolute benefits to the enterprise that cannot be lost. Security, traceability, visibility and control are paramount to running a development organization.

My decision to come work at AccuRev in 2001 was based on the software’s unique ability to provide all the features the enterprise needs, while minimizing the negative impact SCM has on developer productivity. The ease at which AccuRev provides visualization and control over the entire development process was truly a game changer.  And while still a centralized SCM tool, AccuRev was designed to minimize the pain developers felt during their day to day operations.

It’s now 2012 and the SCM landscape has changed dramatically for developers. De-centralized / Distributed development is all the rage now. Version control tools such as Git and Mercurial are providing developers with the experience they demand. Local copies of repositories give unparalleled performance. Strong branching models provide near painless branching and merging. Tools like Git are designed with the developer in mind. Performance, merging strength and extremely in-depth control of all operations are the focus of product features.

All that said, while distributed SCM tools are great for developers and fine for open source development, Enterprises are experiencing chaotic development and losing control in their engineering organizations. So which way do you go? Centralized appeases the Enterprise, De-Centralized appeases the developers.

The answer is both. Now that AccuRev has introduced Kando, Enterprises can maintain all the security, traceability, audit-ability, visibility and process control their business depends on, while giving their developers exactly what they want – to use Git for their day to day development.

AccuRev released for general availability today the first Git integration that seamlessly bridges the world of commercial tools and open source software configuration management.

AccuRev’s Kando, an enterprise security and compliance platform for Git, which was released in beta in January, complements the open source SCM tool with additional functionality, allowing organizations using Git to simultaneously leverage enterprise-level security and requirements traceability.

Kando is the first software development solution that enables Git development shops to add workflow, issue tracking, security, change requirements and other capabilities to the software development process, all while maintaining the flexibility and familiarity of Git environments.

Kando’s ground-breaking enterprise-level security and compliance capabilities allow its users to:

• Comply with enterprise-level regulations, by providing full audit and traceability
• Secure Git with access control capabilities, and support for enterprise authentication via LDAP and Microsoft Active Directory
• Visualize and manage development processes that use Git in AccuRev’s StreamBrowser™ environment
• Support Software Change and Configuration Management (SCCM) through change-based development with AccuRev Change Packages

“Our Kando Beta announcement in January had an overwhelmingly positive response, and it is clear that Git adoption has expanded well beyond the Linux community, into product development and IT,” said Lorne Cooper, AccuRev’s CEO. “Kando can solve many security and requirements traceability problems that enterprise organizations face when adopting Git, and really allows Git an opportunity to scale the development process across enterprise software development teams.”

During the development of Kando, AccuRev established the Kando Technical Advisory Board, comprised of representatives from several enterprise organizations with Git environments. To hear what they are saying, or to learn more about Kando, visit http://www.accurev.com/kando.

The secret’s out – AccuRev is releasing a seamless security and compliance related solution for the Git community called Kando on January 31st. To get a first look at Kando, register here for the webinar on 1/31/2012, at 1:00 PM EST.

You might be asking yourself, “Why in the world would a company focused on providing software development tools to enterprise organizations with mission-critical software development environments produce a solution for an open source version control tool?” I’ll tell you!

Git is increasing in popularity among developers working in small groups or collaborating on open source projects. It’s fast, flexible, and full of developer-friendly features. Git is a great tool for these smaller and more social types of development projects, and based on discussions about Git with customers, prospects, and analysts, it’s clear that there are more cases of enterprise organizations trying to use Git.

But poke around a few blogs, or read a few articles that discuss the use of Git in an enterprise environment, and I’m sorry, but you will uncover a few issues. As one article in BCW discussed, “Git is a version control system with an attitude of collaboration and sharing. There is practically no way you can enforce a specific pattern of access and sharing. If the people who’re using Git don’t want to follow your rules, the tool is not going to help you much.” Let’s be realistic – Linus didn’t originally design Git for use in an enterprise environment!

So, in which cases do enterprise organizations actually use Git?

1. Android Development

If you want to make changes to Android, you’re going to need Git. It’s unavoidable. This means any company creating mobile devices running on Android and working with Android source files has a real business need to use Git.

2. Linux Development

Same as with Android, if your company has a need to make changes to the the Linux kernel, you are going to need Git. Even if you don’t use Git when making those changes, you’ll eventually have to get them into Git.

3. Working with 3rd Party Vendors or Outsourced Teams Using Git

Similar to the Android and Linux situation, if you’re working with 3rd party vendors or outsourced teams who require that you merge your changes into their Git repository, you may be forced to use scripts or bridges to get your changes from your SCM into Git or vice versa, and that’s not a small task.

4. All of Your Developers Love Git

Let’s face it – Git has a cult-like following in the development community. Developers love Git because it’s fast, distributed, flexible, fairly easy to learn, and has a ton of developer-friendly features. It’s developed by developers for developers. Even if you understand the issues Git has with scaling in enterprise environments, it’s difficult to avoid Git when lots of your developers are pushing you to switch.

SCM tools have a profound effect on the day to day life of a developer. These types of systems have either helped or hindered development teams deliver software. SCM systems are like the “hub” of a development team. It’s where teams artifact important work, integrate changes, save important ideas and add features for customers. It’s the center of our development universe!

It’s all about the developers. They need to be free to innovate and get changes out the door quickly. But they can’t if they are stifled by tools that get in the way. Tools need to be able to ENHANCE the software development process. Many people think that source control is just a place to checkin / checkout code. But it’s more than that, it’s where the software development process comes to life. If the SCM system isn’t up to the task of a complex development process, developers can’t innovate.

Sometimes it’s hard to understand that you have a tooling problem, even if it’s staring you in the face. Think of an old pair of trusty sneakers that you have at your house. We all have a pair, they are many years old, beat-up, dirty, torn… but we still wear them. Our feet hurt when we wear them, but for some reason we refuse to get rid of these old sneakers. Until one day (usually after a sprained toe) we decide to buy a brand new pair and after a little breaking in… WOW our feet feel great! Why did I keep the other pair so long?

Software tools are often like this, there is an “if it ain’t broke (too much), don’t fix it” attitude. We often keep tools too long after their expiration date. You’ll hear it from your development team, moaning about the pains of merging code, switching workspaces, checking out … it’s enough to make you cringe. But still we don’t change. Your old SCM is the sneaker, and collectively as a group you and your team have a hard time recognizing when your feet hurt.

The holidays are still several months away, but for software development teams in the travel industry, the “hustle and bustle” of the season is already here.

If you think about the ways you make your business or personal travel plans today, you’ll begin to appreciate the increasingly complex software development challenges travel websites present – and the importance of advanced SCM tools.

Just 10 or 15 years ago, many of us were still making travel arrangements through an agent – on the phone or in-person.  We may have gone online to check out a hotel and moved to another site to check on flights – and yet another to rent a car.

Today, travel websites like Kayak and Orbitz bring together all these consumer options and more – and others include frequent flyer miles, preferred guest points and other information related to our travel plans, often involving outside partners.  All of these variables are changing rapidly and are updated dynamically and in real time on travel websites.  This places greater demands on software development teams – teams that are increasingly distributed across multiple time zones and locations.

Also important to note is that the changes and updates these teams are called on to make are increasingly “business-critical.” A software glitch or a site crash can result in major revenue losses, not to mention the residual consumer frustration and damage to the brand.

More variables, more frequent updates and a more business-critical focus — now magnify all this during the many times of peak travel or weather-related interruptions and one begins to understand why more advanced SCM tools are required in the travel industry today.

Basic software development tools may have been fine for some organizations in travel — smaller airlines or hotels with more basic informational web sites that aren’t designed to process high volume reservations and transactions.  But such basic sites are becoming a rarity in the travel industry, leading more and more travel and hospitality businesses to turn to more advanced software development solutions.

AccuRev’s SCM solutions are designed to handle today’s most complex software development challenges, which explains the growth we’re seeing in travel and hospitality business customers.

So when the holidays draw near and you go online to book a hotel, schedule a flight or rent a car, you can thank all of those software developers that have been working hard behind the scenes to make sure the site is up and running 24/7.

At AccuRev, we’ve noticed a new practice emerging that we call “Issue Based Development.” You may have heard of this term or something like it such as Work Item Based Development or even “requirements traceability.” Whatever you call it, more and more teams are gravitating towards it, perhaps as a result of adopting some of the Agile practices. In this post I will give an overview of Issue Based Development as well as talk about how to maximize your success with the practice.

Issue Based Development is the practice of breaking all work down into individual issues; associating all work products such as code, tests, and documentation with those issues; and managing the progression of work through the various stages of your software lifecycle via those issues.

The benefits of Issue Based Development include requirements traceability, better progress and status tracking, and simpler application of patches from one stream of development to another. These benefits and more will be covered as we cover the individual practices that make Issue Based Development possible.

Track All Work by Issues

Whether you have requirements, enhancement requests, escalations, defects, bugs, tasks, user stories, RFE’s or some other way to describe work, it all goes into issues. You may use a different term to generically describe tracked work, such as work items, but in this post I will use the term issues.

Today, many companies use some combination of Excel, Word, a Requirements Management system, a bug tracking system, MS Project, and an Agile Project Management tool to track “different” kinds of work. In the end though, somebody has to be assigned the work and that work needs to be explained to them in a way that makes sense to them. Why not use issues as the final unit of describing work, whether you continue to use the other systems or not?

Make All Issues Available in a Single Place

Using issues to describe and track all work implies that all issues for a particular project eventually end up in a single Issue Tracking System (ITS). The obvious way to achieve this is to simply move everybody to use the same ITS and to use it for all work-related artifacts, from defects to requirements to whatever else it is that you use. In reality this is difficult to achieve, but is the right goal to strive for, to the extent that you can.

Synchronize Multiple Work Tracking Systems

A good complement to moving as much work tracking to a single ITS system as possible is to synchronize issues from other work tracking systems to a single ITS system. For instance, AccuRev provides AccuBridge and AccuSync to enable you to have all of your work in a single system.

Link Issues to the Work Performed

Once you have all of your work tracked via issues in a single ITS, you are now ready to take the next step, linking all of your issues to the work performed to implement those issues. I’m assuming that you are keeping all of your work artifacts such as code, test, and documentation in your Software Configuration Management (SCM) system. The best way to link your issues to the work performed is to use an SCM object called a Change Package. Some systems may have a similar object called a change list or change set. AccuRev uses change packages.

By linking issues to work artifacts, you can now manage by issues instead of by files. For instance, instead of trying to figure out which files correspond to a set of hotfixes that you want to apply to new development, you can instead refer to the issues that represent those hotfixes and have the SCM system do all the hard work of determining the exact set of changes and promoting those changes from one environment to another.

Use a Stream Per Stage

Having a set of promotion stages is a great complement to Issue Based Development. By having a set of stages corresponding to your development workflow, you can easily see what stage of development various issues are really in simply by querying your SCM system. Doing reports against your ITS are good, but those are generally hand updated. Your SCM system reflects the real state of your code.

A typical set of stages might be “Todo,” “WIP,” “Coded,” “Tested,” and “Done.” You can represent each of these promotion stages with Streams in your SCM system. This allows you to associate Continuous Integration servers which each stage and build versions of your product corresponding to those stages. Imagine for instance being able to build a version of your product that has only tested stories in it.

Another benefit of this arrangement is that it enables “One Piece Flow.” For more on this topic, see my related (http://damonpoole.blogspot.com/2009/09/one-piece-flow-transitioning-from-scrum.html) DIY Agile blog post.

Keep Issues as Small as Possible

Most organizations are good at tracking all bugs in a bug-tracking system, but usually track new work via Word documents which describe very large chunks of work. When moving to Issue Based Development it is important to break these large chunks down into as many and as small issues as you possibly can. This is important for two reasons.

First, it is next to impossible to measure real progress against very large chunks of work. By breaking work down into small issues, it is easier to measure progress by seeing how many issues are fully coded, tested, documented, and ready to ship and measuring the amount of effort associated with those issues.

Second, working on small issues reduces the chances of “issue entanglement.” Entanglement occurs when two issues have changes that are dependent on each other in such a way that they become impossible to promote independently. The ideal is to have one person (or programming pair) work on a single issue at a time and for that issue to be finished in days rather than weeks. The more people working on an issue an the longer they are working on that issue, the more likely it is that the issue will become entangled with multiple other issues.

I’ve often heard people wonder why the SCM system can’t make this simpler. At AccuRev, we are always striving to make Change Packages simpler to use, but the simple fact is that there are limits to how much can be done to simplify entanglement issues. It is generally much more productive to apply the practices in this post which have benefits in and of themselves as described throughout.

Apply the Open/Closed Principle

Large files are another factor that contributes to issue entanglement. Often, frequently used and/or complex classes create large source files. This is similar to the problem of large issues. The larger a file is, the more likely it is that multiple people will be working on it within the same timeframe and thus the more likely it is to lead to issue entanglement.

There is a very useful principle that comes from Objected Oriented programming called the “Open/Closed” principle. This is the idea that software should be open to addition, but closed to change. In its pure form this means that you only add and extend classes and change them as a matter of last resort. A slightly modified version of this would be to only add new methods and to avoid changing them. Also, changing methods should be restricted to rewriting the code such that it adheres better to the principle by being easier to add new methods to and requiring less change to existing methods. This is a very common refactoring pattern.

The side-effect of the Open/Closed principle is that you end up with smaller source code files. The smaller your files are, the less likely you are to run into entanglement and merge issues.

Start Towards Issue Based Development Today

Whether you follow these principles and practices to the letter or not, the more you follow them, the easier it will be to practice Issue Based Development. The better you are at Issue Based Development, the more success you will have with requirements traceability, progress and status tracking, and the application of patches from one stream of development to another.

I’m very excited about our 5.2 release! We’ve completed the move to PostgreSQL on the back end, fully internationalized our products, and added a slew of new features, like per-element security so that you can lock down certain files or directories to specific groups or users.

In addition to moving to PostgreSQL, we’ve taken advantage of its capabilities to increase performance in a few key areas such as update, especially when using cross-links, populate, and more. And as a first step towards fully embracing 64bit servers, we now support 64bit CPUs on Linux servers. You can look forward to additional performance boosts as we exploit the capabilities of PostgreSQL in the future.

Haven’t upgraded in a while? Check out what else is new! To get a full flavor of all of the new functionality we’ve added recently, check out recent webinar which highlights what’s new.

If you haven’t tried our plug-ins in a while, we’ve put out new releases of our Eclipse and Visual Studio plug-ins for all versions of AccuRev. Eclipse and Visual Studio plug-ins will work with older releases, so no need to wait for 5.2 to start benefiting from the new features in the plug-ins.

We’ve fully integrated the Web UI into the plug-ins so you can now use all of the following functionality: version browser, annotate, stream browser, and all of AccuWork. Also, in 5.2 you can now access the Web UI with a single click from most screens. For instance, if you want to send somebody a URL of a file or diff, you can just click the “copy to clipboard” button in the toolbar and then compose an e-mail to somebody and paste in the URL. When they click the URL, they will go right to that file or diff. And the Web UI now allows you to print any table that it supports.

While we are in the process of making the new functionality in the Web UI available in all of our UIs, this is a great first step to quickly link to the new functionality.

A Few More Details on What’s Available Via the Web UI
If you haven’t used the Web UI or are using an older version, you may not have heard that there is an entirely new version browser in the Web UI and for AccuWork users- AccuWork on the Web has been completely revamped. For instance, it is easy to search by keyword and you can do drag and drop query editing.

5.3 and Beyond… Here We Come!
The database and internationalization are two very important infrastructure changes which will accelerate both our engineering velocity as well as our market penetration. With our upcoming 5.3 release we will be re-introducing our quarterly pace of releases as we had with our 4.8 and 4.9 releases. With the switch to PostgreSQL and an internationalized code base complete, you will see the release of new functionality faster than ever before. In preparation for 5.3 and beyond, we’ve just launched a new product survey as one of the many ways we collect product priorities. Make sure to look for it and fill it out. It is one of the best ways to have a significant impact on our product roadmap. If you haven’t gotten an e-mail yet, you may not be in our survey database. Just ask your AccuRev administrator for a link.

I mentioned on this blog last week that AccuRev had big news coming that would mark the next phase in the evolution of SCM – the first big leap since AccuRev introduced the stream-based approach. Well, the big news has arrived with this week’s launch of AccuRev SCM Version 5.2. AccuRev Version 5.2 arms developers with one of the few SCM solutions on the market today that offers streams and enhanced enterprise performance, reporting, security, and scalability through the postgreSQL relational database we’ve embedded into our system.

With this release, AccuRev has also introduced AccuWorkflow 2.0, a collaborative process-centric governance tool that enables automation and compliance with an issue-based lifecycle workflow directly within AccuRev. AccuWorkflow is especially valuable for developers of software solutions for highly regulated industries such as financial services with its stringent requirements for SOX compliance.

AccuRev 5.2 is available immediately.  Check out the press release here  for more details, or see us demo 5.2 at Agile 2011.   You can also check out a brief podcast we put together that examines the major issues and challenges today’s enterprise developers face and how we’ve attempted to help solve those challenges with AccuRev 5.2.  Click here to listen in.

For more info, check out What’s New in 5.2.

There are a handful of SCM best practices that can optimize continuous integration.  This post will look at:

• Establishing a staging and isolation hierarchy
• Automating builds at all stages in the hierarchy

Establishing a staging and isolation hierarchy for optimizing Continuous Integration

Proponents of continuous integration commonly suggest branching as little as possible and having developers work directly from the mainline as much as possible. However, this approach has several difficulties:

• It puts the stability of the mainline at risk
• It presupposes that traditional legacy branches are the only available isolation mechanism
• It decreases the flexibility and agility required for fast iterative development

With modern SCM systems, a better approach is to implement a staging and isolation hierarchy for the development process. A staging and isolation hierarchy uses objects in the SCM system to represent the dependencies between development groups and process steps. For example, you may wish to model the following teams and activities:

• Release engineering
• Quality assurance
• Product engineering
• Component engineering

Each team or activity is assigned the equivalent of a private workspace (variously called “streams” or “branches” depending on the SCM system). Each team then receives the same benefits of private workspaces that individual developers receive.  With a staging hierarchy, changes move from less stable configurations to more stable as they are tested and deemed “good” for the next level. This allows the code to be stabilized as it gets ready for release without developer downtime. It also allows additional separation for each team if needed, so that the team’s changes can be integrated and tested before the components are integrated together.

In this figure, there are four development teams as well as an area for accepting third-party code drops.  The teams are located in different geographical areas. The hierarchy represents the normal flow of changes through development from stage to stage. In the example of the above figure, changes provided by the GUI product engineering team in India flow from individual developer workspaces (not shown for brevity) to the GUI stage, where they can be continuously integrated and tested. Mature changes then flow to the UI_int stage and on to the QA and Release (Rel) stages, again being subject to continuous integration and testing at each stage. The web development team in Austin picks up well-tested changes from the UI_int stage and uses them as the basis of their development work; when the web changes are mature they can be pushed up the hierarchy and subject to broader testing in the UI_int, QA and Rel stages.

Using a development hierarchy provides more opportunities for check-pointing. Every change introduced into the system is a potential source of failure, and thus a potential checkpoint. If a change proves to be unstable, you can return both the source stage and the destination stage back to a previous checkpoint. By contrast, mainline development only offers you a single opportunity for check-pointing, specifically, the state of the main codeline itself. Unless your development process includes “freezing” the mainline for a long enough period to build, test and otherwise validate, the chances of isolating and check-pointing at an appropriately fine level of code granularity are slim, making any available checkpoints stale and of limited utility.

Automating builds at all stages in the hierarchy

In order to give developers prompt feedback about the changes submitted, the code must be built frequently, ideally several times per day. A continuous integration server such as CruiseControl, CruiseControl.NET or Draco.NET can be employed to automate this process. The continuous integration server periodically polls the SCM system for changes, populates the changes to the build server, initiates the build process, and reports the results of the build and unit tests.  It is important to note here that the continuous integration server utilizes the existing build scripts and build environment to execute the build. For example, if make is used to compile and link components written in C, then the continuous integration server will call the makefile to initiate the build process.  Because the continuous integration system uses the existing build, it is important for development groups to devote time and effort to:

• Making the build as fast as possible,
• Building automated unit tests and
• Including unit tests as part of the build process.

Spending time on these items, even if it involves some rework of the build system to make it more compatible with a continuous integration environment, will improve not only the build process but the overall quality of the software release.

When utilizing continuous integration, it is crucial to communicate the results of the builds to the entire development team. Continuous integration system planners should consider a scalable communications method such as e-mail notification or an internal website to display build results.

Continuous integration servers such as CruiseControl come with built-in web reporting that can be easily customized, so that build results can be displayed on LCD panels in common areas at geographically dispersed locations. In this way, team members can easily see and respond to the build results and reduce the “fix latency” often encountered with nightly or weekly integration build approaches.

Goals of Continuous Integration

If a primary goal of continuous integration is to improve software quality and reduce downstream build and test failures, then perhaps the single most important SCM best practice is to enable automated local developer builds. To this end, developers should be provided with access to a local build environment that mirrors the production build environment, including:

• Database schema
• Configuration files
• Environment variables
• Shared libraries and files
• Known compiler/linker/runtime environments

This minimizes the “it compiled fine on my machine” syndrome that plagues many development efforts. Once the build environment is available, every effort should be made to automate the local build, so that each developer can build easily and quickly without having to perform a series of manual steps.

Ideally, the developers should have access to a private build area that uses the same tools and configurations as the production builds. As part of configuring a local build, developers should be trained to take several steps to ensure that local changes will have minimal negative effects once shared with other developers and teams. These steps are as follows:

• Updating their private workspace to obtain the latest shared configuration code
• Merging conflicting changes from the shared configuration to their private workspace

By ensuring that developers have known, good build environments, and that they have the latest code with all conflicts resolved, there is high probability that the local build will succeed. If the local build is successful, a continuous integration build that includes the developer’s changes should also succeed.

Shifting to Continuous Integration and Frequently Updating Code

Traditionally, developers tend to put off sharing their changes, sometimes for days, because they don’t want to affect other people too early, or don’t want to get blamed for breaking the build. Unfortunately, this strategy tends to backfire and typically leads to more problems in debugging larger sets of changes after a build breaks. Independent of whether continuous integration is being employed, encouraging developers to build, test, and share their code in small “chunks” is a simple and effective way to improve team collaboration and reduce costly broken builds.

The importance of frequently updating the code in the SCM system is amplified when organizations move towards a continuous integration model. Continuous integration represents a paradigm shift for software development, emphasizing communication between developers about what changes have been made. By breaking down tasks into small chunks that take several hours to complete, developers can “commit” or “promote” their changes frequently and receive immediate feedback about the quality of the changes through the continuous integration build and unit test results. As teams get accustomed to this new approach, developers also gain a sense of progress by seeing not only their own changes build successfully, but also seeing the changes that other developers have made available.