You spent the last 2 weeks working on 10 user stories for you sprint. Your team has completed 8 of the 10 user stories, and now it’s time to show a working demo for your sprint review. Sounds like a typical scenario right?

Here’s the problem, the 2 remaining user stories have partial work associated with them. That code is in your source control system, and in order to correctly demo or ship what was accomplished in your sprint, you’re going to have to do some work to create a “done” product. You will have to subtractively merge those changes, or put them on another branch, then re-test the application, redeploy.. check the changes again etc. There is still a lot of work to finished in getting this sprint to “done.”

One way to get around this is to associate user stories with code changes in your SCM tool.  AccuRev already creates a link between your user stories and code in the form of change packages. Creating that link between the user stories and your code is the first step. But using a powerful story-driven process out of this linkage is the key to “getting to done” in an Agile environment.

Your user story will move through different stages as you move through your process. One user story may have these states:

DEV- Currently in progress with dev team, or “coding”
QA- Coding finished, ready for QA to test
UAT- QA finished ready for user acceptance testing
PROD- Tested by QA and users, ready to ship

However, while  user stories move through this process, the code doesn’t follow. Traditional SCM systems are designed around single branches and a long and lengthy merge processes. Fast paced Agile environments expose those limitations as development teams struggle to ship code to customers at the end of iterations.

During that iteration, testers may want to test completed user stories, but need a stable configuration to do so. If you’re using a traditional waterfall branching and merging based SCM tool, you may end up with some user stories ready for QA, and some still in the DEV stage, creating a poor testing environment and broken builds. As a result, developers often delay committing code so changes can be implemented, tested, and passed to QA before they integrate.

During your retrospective of the previous iteration, you may decide that parts of your process need modification. For example, you may decide to add a code review process right before testing. Your SCM system has to be flexible to enable this new step without time consuming tasks, like re-writing scripts.

SCM and an Agile Process

The trick is to have an SCM system that can help you manage and enforce an Agile process. Code should be able to follow the same process user stories follow. If a user story is in QA, all of the code needed to test that issue should reside in a QA configuration. The same goes for DEV, UAT, PROD, or any other process that is part of your environment.

A stream structure is an easy way to  process change – streams can be added to adjust your process. In a few clicks, you can add a code review between a DEV, and QA step.

Agile and SCM: Avoiding Agile Merge Hell

As you start to scale Agile, code and user stories have to be merged more often. Sometimes changes my flow from one organization to another. This means that you will need to take code from one team, merge, integrate and test those changes with everyone. Each team needs to be able to work on their own schedule, this means that if multiple teams want to work on different sized iterations they can. In addition they can deliver changes as they need to on a regular basis, independent of the the other teams.

The problem here is that to do this in a traditional SCM system, you would have to merge these code changes daily for them to be of any use. You also still need to keep visibility into what stories are shared between teams, because delivering changes from user stories that are not completed in a sprint would be disastrous.

Typically, the type of configuration people use for this is a single baseline or “trunk” methodology, where all changes from each team are delivered to trunk and pulled from trunk as their iterations are completed.

Working within an Agile context, your teams will have to deal with these branching and merging issues. But there are other problems that can happen when you use this methodology:

1. Delivering 2 weeks worth of changes only causes isolation among teams.
2. It’s too hard to pick out each user story as it completes from our codebase.
3. Figuring out the dependencies of those user stories is complex.
4. Being able to identify what changes came from each branch is impossible.

This “baseline pollution” is not scale-able. You can get around this by using a development hierarchy, and manage the relationship of dependencies between branches. This could also include process steps such as integration, quality assurance, and code reviews.  A separate code configuration can be used for each step and user stories could simply be drag and dropped between each team, state or branch instead of a merge.

Doing this will increase code stability. As a completed user story is pushed from one stage to the next, the particular change, as well as the system as a whole, reaches a higher level of maturity. While many traditional SCM tools do not easily support or surface a development hierarchy, AccuRev supports the creation of a hierarchy, gives visibility into the changes at each stage, and enables straightforward merging between stages.

We recently produced a webinar here at AccuRev, co-sponsored by Rally Software, about the Top 10 Factors for a Successful Agile Implementation.  #10 on that list was the Executive Commitment.  We all know that having an executive team committed to the success of a project of any kind is important, but in the Agile world it is even more critical.  Adopting Agile as a new methodology is a major shift in the way of thinking of how your business is being run, for the better.

As Agile adoption starts to take hold in your organization it can follow two different paths – from a motivated development team on up, or from an executive edict on down.  Either way you have decided to make a move toward Agile adoption, one of the very first hurdles you will encounter is executive buy-in.  Many changes are in store for your organization and to insure success you’re going to need to have that executive commitment.

First and foremost, let’s make sure the executive team has complete sponsorship of the Pilot project. While I will discuss selecting a Pilot team in a future blog post, you need to make sure that management is behind this launch. Management needs to be completely committed and willing to smooth any resistance that you may run up against as that pilot process starts to take off and Agile practices start to take hold.

Next is the area of trust and respect.  The people taking part in the transition need to do their best. There’re going to be a lot of changes happening within in the organization: the way people think, the way people function, the way teams interact.  Management really needs to be committed in changing to Agile development, and while there are going to be some pains and adjustments necessary, management will trust and respect that people on development teams are making the best choices for the organization.

Trust and respect is important due to the fact that Agile practices cause a major shift in the ways of thinking.  You’ve probably heard mention of the cultural change that Agile is going to impart on your organization. Executives need to be aware of a major shift in the way people think and the way people do business.

To expand on that just a little bit, it is necessary to ensure that the entire organization supports the transition to Agile development.  I’m not talking about just the development teams; we need to make sure that management understands that it’s going to impact the customers, and for the better. Agile is going to make the organization more productive, more effective, and more efficient, but it is going to cause changes and growing pains as we move towards that process.

One other area that I wanted to bring up briefly, is that going Agile does require an investment.  So management needs to understand there will be upfront investments in transitioning to Agile development and in turn, reaping the benefits of Agile.  That being said, rate of return on your investment can be very swift.  I’m tempted to say “will” be very swift, but obviously it’s going to depend on the successfulness of the project.

To understand some of those returns is to really understand the business benefits and value as well as making sure executives understand those as well, because there is a lot of value to the organization by adopting Agile. It’s not just making better, faster developers, obviously.  We’re talking about faster time to market.  Because we’re developing and shipping smaller increments of high-priority code we’re able to get that code to market much more quickly. That is obviously going to mean increased revenues by being able to hit the market need more quickly and get EXACTLY what the customer wants out the door. We’re going to be able to achieve an increase in revenue based on those features and those enhancements that are going out the door.

All of this because of the way the Agile development process is done and the way testing is baked-in and moved closer to the development cycle.

So all of those features that are getting to market quicker while increasing our revenues are going to be higher quality, as well.  Overall that’s going to greatly increase customer satisfaction as to the type and the software features they’re actually seeing based upon our Agile development.

Now that the executives are on board… how about we take a look at picking the Pilot?

As an AccuRev blog reader, you’re welcome to attend our upcoming free webinar that will discuss the intersection of Agile development and governance:

Attend this Webinar and learn:

How Agile processes and compliance can coexist in harmony

What pragmatic ALM is and how it can help you solve today’s business challenges

How to manage multiple processes dependent on project requirements (Waterfall, Agile, etc.)

Best practices for optimizing tools and processes for both software development and release management.

When: Thursday, December 4 at 1:00 PM EST

Register: The Business Case for Pragmatic ALM: Agility with Governance

by Lorne Cooper, CEO, AccuRev 

A Top 25 WordPress Blog Post of the Day

In Part 1, we looked at the sins of Building a Weak Team and Ignoring the Future.  Once you’re committed to hiring talent and building a forward-looking process, there are two more sins to avoid.

Sin #6: A Long Tail After Development is Complete

We constantly see organizations where the time from “requirements-in” to “feature freeze” is half the time to “solutions-out.”  That 50% “tail” on development provides no direct value to customers, but is there because of limitations in our development process.

Sometimes that tail is twice the length of the initial development. 

Ironically, while the long heavy tail is grown to protect the organization from delivering bad product, it eventually decreases response time until it becomes an instrument in the organization’s death.

But many industries have a long tail, don’t they?  Surely it doesn’t hurt the construction of bridges that the plans are completed long before the bridge is built.

For forty years people have tried to make analogies between software development and more mature engineering disciplines, say, like building buggy whips.  The designer of the buggy whip might make a sample or two to try it out on his children, then send it off to the factory, where a slew of Industrial Buggy Whip Engineers would spend months figuring out how to sew the damn things, buying up horse-hair, and hiring immigrants to sit at sewing machines. A year or so late, Pony Express delivers the new buggy whip to general stores all over the Grand Trunk Railway line. 

We can all certainly learn a lot from that.

Now try telling your customer that the problem she’s been working around is well understood, and in fact fixed, but she’s not going to get it for three months.  She’ll be looking for alternative suppliers before the screen door has hit your butt. 

Everything moves too fast to use that tired old process.  Frankly, I use to blame computers.  Now I blame the Internet. 

When you take a hard look at the time it takes to get completed product out the door, you’ll probably find it dominated by two things: testing, and fixing bugs that the testers find.  Both stem from the problem that you’re not doing enough testing during the development process, and you’re trying to test too much at one time.

I’m no fan of Hawthorn-effects, or what the politically incorrect might call gimmicks, like pair programming or stand-up meetings, but projects that use an Agile development with automated regression test processes are certainly doing more right than wrong. 

Is it really so hard to manage by tasks/issues, normalize them to a small unit of work, merge in just one set of changes into each build, and test just that perturbation?  What if you knew you would consistently get better code out the back of the process, and therefore make your organization more responsive to your customers?

Your development process must minimize latency, instead of trying to maximize bandwidth.  Don’t miss the changes in requirements and environments that produce opportunities for better products, or sooner or later you’ll end up locked in a suicide pact with a project no one wants.

Sin #7: Complacency

Calais’s stone walls were so thick the city had no fear of invaders, until they showed up with cannons.  Napoleon’s Old Guard was undefeated in battle after battle, until they were cut down by Wellington at Waterloo.  Big Blue’s OS/2 would crush little Microsoft’s Windows product.  Big Microsoft’s Live Search would crush little Google’s search engine. 

Hubris on ice will leave you with a heck of a hangover.

That brilliant team using that wonderful process you put together last year, is just a few months of complacency away from dysfunction.  People move and, thanks to the 13^th Amendment to the US Constitution, may leave their jobs and there’s little you can do about it.  Environments change and architectures become outdated.  The marketplace changes and requirements change with them.

One VP Engineering I know told me he asked his off-shore application development partner to give him 110%.  They did, but it turned out to be turnover.

Your process needs to be constantly monitored and upgraded, because everything in the process is changing.  If your offshore partner has high turnover, their part of the process might require more code inspection and knowledge transfer.  If your biggest customer can’t use your product until it supports their new environment, you need to put a special team on building a custom version of the product, and then merge that back into main development.  If you have to replace an old veteran with a rookie, then you might create new integration points and a more rigorous test protocol.

Recognize that everything will change, and the assumptions under which you built your last success are no longer valid.  Make small process changes early and often, or in the long run the big changes will be made by your replacement.

Stirring Conclusion

Managing a software development organization is a very challenging task, as tough as playing a violin with your toes, and less fun. 

I don’t mean to trivialize important management functions such as customer management, budget and project tracking, running a good meeting, and designing effective compensation structures. And by all means, hiring someone to spend their time doing all of that.

But that won’t protect you from the seven deadly sins.  Only you can do that.

I’ll finish with the Four Virtues, which are the antidote to the Seven Sins:

1. Hire great people or don’t hire.
2. Plan to build a product that will have a long life.
3. Keep your process nimble, and deliver less, faster.
4. Keep improving everything.

Yesterday I was a panelist for a Webinar on agile tools, focusing on software configuration management (SCM), build and software process automation (SPA), the latter term referring to the set of defined, repeatable and measurable automated development workflows that engineers use to transform requirements into shippable software products. Contrary to what I’ve read about the disdain that some agile devotees have for tools, most of the attendees were hungry to know what features their SCM tool should have in order to support agile software development and SPA. Here are some of the highlights, and of course, my take on why I think AccuRev is the best tool for agile software process automation.

There are five key feature areas that an SCM tool needs to support in order to be ready for agile:

* Support for flexible process models

* Continuous integration support

* Support for issue-based development

* Efficient branch and code management

* Private version controlled developer workspaces

Let’s take a look at each of these in turn.

* Support for flexible process models. Agile is often one of several processes being employed within a software development organization. Unless your SCM tool is flexible and process-neutral, you will have a hard time implementing agile (say, for product development) and more traditional processes like waterfall (for example, for product maintenance work) in the same SCM tool. AccuRev streams are a natural way to model any process, and thus are a good fit when agile needs to coexist with other development processes. As for software process automation (SPA), AccuRev streams again are a great fit, since they enable users to model any arbitrary stages of code transformation that a development team sees fit to define as part of their process. By adding triggers and workflow to a stream hierarchy, teams can implement SPA directly in AccuRrev.

* Continuous integration support. Continuous integration is one of the core process elements associated with agile development. By building and testing frequently and acting on the results of tests, teams can uncover defects or test gaps earlier in their development cycle, saving time and money compared to such discoveries late in the cycle. But continuous integration goes beyond just testing the nightly build. With multi-stage continuous integration in AccuRev, code is automatically promoted up the stream hierarchy into more stable configurations as it passes tests. At each stage, continuous integration takes over to build and test, typically with a wider scope of testing as the code nears the release stage. Legacy SCM tools make this type of automated integration factory somewhere between difficult and impossible due to the complexity involved in setting up the hierarchy and in automatically moving and merging code as it flows up the hierarchy.

* Support for issue-based development. Apparently there is a lot of contention about the need for filing issues and defects in agile development. This has puzzled me greatly. While I’m in favor of developers identifying and fixing issues as they are discovered, you lose valuable process information when a defect or enhancement ticket is not filed and later associated with a code change. Without an issue that describes what the problem was, someone looking at the code changes for audit purposes or for group code reviews is at a disadvantage. Why was this code change made? Is it related to other changes? How long did it take? Was it done to fix a bug or to add a feature. In AccuRev, issues either in the integrated AccuWork issue tracking system, or in a 3rd party issue tracking system, can easily be associated with code changes via the AccuRev Change Package mechanism. This establishes basic traceability between issues and the code changes that developers make in order to satsify the requirements of those issues. Issue-based development is well-defined, repeatable and measurable – all hallmarks of good software process automation.

* Efficient branch and code management. Any time you’re working on more than one project, you need to isolate that project’s code from other projects. With agile and multistage continuous integration, even a single project requires multiple code lines in order to separate in-progress code from unit tested code from system tested code that is ready for release. If an SCM tool makes branching, merging and labeling difficult, teams tend to practice branch avoidance, which I sometimes like to call “fear of branching.” This is a classic example of letting a tool dictate what processes you can implement. In AccuRev, streams replace branches as the mechanism for isolating codelines. Since streams are represented inside of AccuRev as data separate from the actual file versions, creating streams is fast – really fast, like, a second or two – and managing a system with hundreds of streams spanning multiple projects and processes is easy.  For continuous integration, AccuRev snapshots and time-based streams are also fast and easy to create and manage, and give users a straight-forward way to “label” an interim or milestone codeline without having to place markers in thousands of source files.

* Private version controlled developer workspaces. Software developers are the heartbeat of any engineering organization. Executives at any development shop will tell you that hiring talented engineers and keeping them well-tooled and productive is the single largest challenge that they face. For agile, this is even more of a challenge, since coding cycles tend to be shorter, and thus anything that gets in the way of individual or team productivity tends to have a greater negative impact on the project. Private version controlled workspaces like the AccuRev workspace model improve productivity, since they enable developers to work in isolation (while they are ‘heads down’ coding). Private workspaces in AccuRev also give developers full SCM capabilites in their workspaces without the need to share in-progress code prematurely. By using the ‘keep’ operation, developers make safe copies of their work in the AccuRev repository, and later can ‘promote’ the code to an integration stream to combine their work with that of their teammates. Individuals are more productive in this way, and if continuous integration builds are frequently testing the integration stream, so are teams.

In a nutshell, agile requires tools, and these tools need to support different modes of operation than with other processes. SCM can help or hurt you in setting up and executing an agile process, so these guidelines are a way to help you get your SCM tool ready for agile - easy of course if your tool is already AccuRev!

If you’re interested, you can view the webinar recording.

Is Your Software Development Environment Agile-Ready?
Free On-Demand Webinar

A long, long time ago I started life (okay college life) as an Engineer. Electrical. Analog. For having grown up with always having a computer in my room I was about as far away from the digital world I could get and still be technical. Computing was easy. Oh pascal, if only you could see me now. Those were wild days, staying up until all hours of the night sitting in a basement computer lab trying to finish an assignment. When I wasn’t at classes or getting into a variety of trouble, I would spend time with my roommate, a Mac user. He’d mock my Amiga; “Do you really need two buttons?” I’d mock the Mac (pre-classic) “Hey I’ve got 4096 colors” but we’d agree on one thing, both were better than that loser IBM machine two dorm rooms down the hall.

Then came real life, I fell into the digital world, and found myself back in school getting a more formal digital education. I found myself in Design Patterns, listening to people around me furiously debating the merits and needs of singletons, factories and observers.

Then came Java, and the arguments continued. Performance, idiosyncracies of langauge (both C++ and Java) etc., etc.

Then came Open Source. Don’t get me started with open source. If the OS mafia reads anything I write they’ll try to shutdown our website.

Now we have Agile. And Lean Development. And Continuous Integration (CI). I’ll lump them together because they are the current arguments I hear today. Yeah, arguments. Developers become elevated and yell out “You’re not really agile!” I read books on what it means to be Lean, and how to identify if you are Lean. I find CI writers muttering under their breath that any build process that occurs hourly isn’t truly CI.

They’ve lost focus. Or more importantly their focus should not be your focus.

I wasn’t hired because I was a Design Patterns zealot. I wasn’t hired because there was a need for an expert on Lean Development. I wasn’t hired to support code in the OS movement. I was hired to produce a product. Our customers want our product, they want it yesterday, both features and fixes.

So what’s the best way to deliver value to the company and customer? To be honest, I may not know the best way, but I know we will do our best to do whatever it takes. I and my coworkers strive to find reasonable solutions in a reasonable time.

But the zealots cry out and tell you what you are doing wrong, not doing right. What they don’t understand is that if they want to be heard they have to show the value in the changes. You need to give the agile concepts breathing room. This is what really encourages adoption. Agile zealots need to understand why there is resistance, understand people’s concerns and address them, and believe me they will embrace the positive elements.

On the other side, why listen to these zealots? Because if you can get past the worship and hype, there is value in what they describe. If you listen, you can figure out what elements will be useful.

Agile really is an evolutionary model. Agile isn’t about anarchy or cowboy developers, but about adopting processes that improve on process management. Agile really helps break logjams where problems seemed intractable or a great unknown, places where development has struggled for decades over thinking problems only to have their solutions fall flat and disappoint customers.

Lean Development has a great focus on continuous refinement. Focus on problem areas and work to make it a little better. After you’ve done this take another look and make it a little better somewhere else. Very quickly the improvements add up.

And Continuous Integration keeps simple problems simple, resolving them in a timely manner. CI keeps developers from sitting on problems further compounding integration, testing and deployment.

There are values that are worth a look in all of these movements. Take what they give you, find the value and use it. Continuous Integration makes your life better. Problems are found (and hopefully fixed) when they happen, not weeks later. You are getting continuous feedback on the health of your project, not monthly or weekly feedback. If your life is better you can spend more time providing value to your company and your customers. If you deal with less pain you have more time to solve the problems you should be solving. And adding the features that your customers are waiting for and wanted yesterday.

What are you waiting for? Don’t be discouraged thinking you aren’t using CI ‘correctly’. Take elements and make them work for you, and the value will come quickly.