In the last post I defined two Agile software development best practices I believe provide value to a wide variety of development teams.   Here I define three more practices that I believe are also important when transitioning to Agile Software Development; collocation, unit testing, and refactoring.

Best Practice for Agile Software Development: Collocation

Collocation is simply having everybody on a cross functional team in close proximity to each other. This compounds the coordination benefit of cross functional teams. This is orthogonal to outsourcing. Whether you are outsourcing or not, collocation only refers to whether a particular cross functional team is sitting near each other.

Best Practice for Agile Software Development: Unit Testing

Unit tests are simply tests that exercise small amounts of isolated functionality. That is, if you have a function that adds two numbers, instead of depending on running a user function that eventually calls the function, exercise the function directly. This often requires the use of mock objects that pretend to be things that the function needs in order to test the function in isolation from other functions that it depends on.

The cost of unit tests is in writing the tests themselves and refactoring code as new functionality is introduced to keep the unit tests testing at the right level. The benefit is that you can easily test changes quickly to find simple problems before doing more thorough and slower testing. It also provides a good safety net for refactoring, gets developers more involved in testing, and usually improves the design of the software.

Best Practice for Agile Software Development: Refactoring

Refactoring is the practice of continuously improving the usability, maintainability, and adaptability of code without changing its behavior. That makes it much easier to add new and unanticipated functionality. Refactoring has the disadvantage that it takes extra effort and requires changing the code. Any change has the potential to reduce the maturity and stability of the product, especially if you don’t have adequate testing in place. That’s why refactoring is usually paired up with unit testing and together these are frequently combined with continuous integration.

While many experts push for Agile adoption throughout organizations, a different trend within organizations is emerging.  Organizations are taking smaller steps towards Agile by focusing on specific Agile techniques and applying them to their individual development processes. By adopting certain techniques, organizations still receive some benefits of Agile without “going Agile”.  After all, it is possible to do Continuous Integration without time-boxing and still reduce integration problems.

Getting Benefits of Agile Software Development

For those interested in gaining the benefits of Agile through smaller and more targeted techniques, there are a number of possibilities.  Below are recommended readings on Agile techniques, each focusing on one specific area, while detailing the processes and outlining the benefits.  Here is a great place to start when looking to gain the benefits of Agile software development.

User Stories Applied: For Agile Software Development, by Mike Cohn.

The Agile equivalent of requirements is User Stories, and this book is the definitive guide to creating and using them.  Readers will not be disappointed with Mike Cohn’s practical advice on User Stories, as the information offered is easy to read and apply. Valuable topics discussed include why to use User Stories, when they are appropriate, and how to monitor progress.

Continuous Integration: Improving Software Quality and Reducing Risk, by Paul M. Duvall, Steve Matyas and Andrew Glover.

This is a great resource for Continuous Integration. The authors stress the importance of integrating early and often using CI practices and provide detailed results of a successful CI implementation (including reducing risks, better visibility and reducing repetitive manual processes). Another important element of this book is its companion Web site, www.integratebutton.com, which provides updates and code examples.

The Art of Unit Testing: With Examples in .Net, by Roy Osherove.

This is one of the best on the topic of Unit Testing, partly because of RoyOsherove’s experience and passion about the subject, and partly because of his practical delivery. His book covers important areas of Unit Testing, including writing maintainable test code, code review and adopting Unit Testing in an organization.

Refactoring: Improving the Design of Existing Code, by Martin Fowler, Kent Beck, John Brant, & William Opdyke

This book offers important technical insight into the process of changing code to improve internal structures without changing external behavior and is a highly recommended resource on refactoring and improving design of existing code.

To get more benefits of Agile without going Agile, try JUnit Recipes: Practical Methods for Programmer Testing by J.B. Rainsberger, and Implementing Lean Software Development: From Concept to Cash by Mary and Tom Poppendieck or visit AccuRev’s Agile Resource Center.

More Agile resources

AccuRev is hosting a free Webinar on “Emerging SCM Best Practices for Agile Development” on Thursday, February 5 from 1:00 – 2:00 PM EST. The webinar will introduce the unique demands that agile processes place on legacy SCM tools and ways to build and automate an efficient Agile development process. Damon Poole, AccuRev CTO, and Uttam Narsu will be presenting.

Mr. Narsu, an industry expert on software configuration management (SCM) best practices and former Forrester/Giga analyst, will discuss the most important aspects to consider when applying SCM best practices to an agile world.  Mr. Poole will discuss how Multi-stage Continuous Integration can solve some of the underlying impediments to a successful Agile development environment.

“SCM is critical to Agile success. If your SCM tools don’t provide integrated and seamless support for Agile, you won’t get widespread adoption of Agile development,” said Narsu. “Development teams require brutally efficient SCM tools, but the tools must still be issue-based, must have support for flexible process models, and enable efficient branching, merging, and refactoring. The SCM tool should also provide private workspaces and assist continuous integration.”

Mr. Narsu has worked with hundreds of clients using wildly differing software development environments. While clients who succeed with agile development have strong SCM practices, the best practiced agile SCM: transparently integrated, flexible, and collaborative.

Specific topics discussed will include:

• Typical SCM obstacles to Agile success and how to avoid them;
• Key Agile Process requirements for SCM products and specific use case scenarios;
• Challenges with continuous integration, and how Multi-stage Continuous Integration delivers value and how to adopt it today; and
• Key SCM metrics for delivering on Agile development goals.

Register here for this free webinar taking place on Thursday, February 5th from 1:00-2:00 EST: Emerging SCM Best Practices for Agile Development.

by Dave Thomas

Your team has been coding a feature for a few days, across dozens of files, and everyone is excited with progress.  Deep into development, you find an interesting 3rd party library that may simplify your work and possibly take the team’s feature to the next level.  But it will require adding new files, moving some directories, and refactoring some critical code.  However, you’re not ready to commit your current changes because they are unfinished and will surely break everyone else.  Your changes need to be saved before prototyping in case of a rollback but also be integrated with prototype development.  And what if the new library is a bust?  If you start co-mingling the library integration with unfinished code, the potential revert process will be a complete nightmare and waste of time.

Private Versioning. With AccuRev’s private workspace, you can always commit your changes early, often, and safely without sharing amongst your peers.  But in this case you have two logical development efforts, the 2nd effort depends on the first and both need commits to preserve evolving changes.   How do you keep them cleanly separated and continue to work on both in parallel?

Stream Inheritance.  AccuRev streams have an intriguing and very powerful feature called inheritance.   Similar to how an OO subclass implicitly inherits methods from parent and grandparent classes, a child stream implicitly inherits versions of files & directories from parent and grandparent streams.   Taking advantage of inheritance, we can use streams to independently manage logical changes and cleanly maintain change dependencies without physically co-mingling files.

The Pattern.  Prototyping changes without co-mingling files can be done by simply creating a series of ‘personal’ or ‘private’ development streams (though, they are just regular dynamic streams). This pattern will create a “Feature”, “MyDevelopment”, and “MyPrototype” stream sub-hierarchy.  See Picture.

click to enlarge

From your Integration stream (or equivalent), start by creating a “Feature” stream that will collect all changes from your entire team.  [See related blog, Stream-per-Task Pattern].  The majority of team members may have their workspaces directly from here.  Next, create a child stream called “MyDevelopment” that will track your personal ongoing development activity.  Finally, create a grandchild stream called “MyPrototype” to track changes that will be discarded or retained depending on the level of success.

The prototype development activity is committed, shareable, integrated, and yet cleanly segregated from both active feature and private development.

The “Feature” stream is the collection point of all in-progress development activities for the given feature from the entire team.  Developers will promote here frequently possibly kicking off an automated build with success/fail notification.   The “MyDevelopment” stream provides a collection point for all of your personal development changes.  This stream may be considered a “private” development stream simply because no other developers will likely use it – set a stream lock to be sure.  A lightweight Continuous Integration build (i.e. compilation only) may be performed on “MyDevelopment” for sanity sake or just compile and promote as a practice.  The “MyPrototype” stream is a collection point for all prototype changes. Even as new changes are promoted to “Feature” and “MyDevelopment”, the “MyPrototype” stream will automatically incorporate those changes (via inheritance) and the prototype developers will merge changes as necessary. The prototype development activity is committed, shareable, integrated, and yet cleanly segregated from both active feature and personal development. Also, by using a stream for prototype work, multiple developers can contribute and collaborate. If the prototyping work is deemed successful, the files can be promoted to “MyDevelopment”. If the prototyping efforts don’t work out – no problem – just remove the “MyPrototype” stream and re-purpose the workspaces.

The beauty of this pattern is that it isolates development activity by purpose without co-mingling physical file changes.   It lets the prototype developers go nuts and shoot from the hip while the regular feature developers (with the deadline!) work unimpeded and without fear of rampant changes — and everyone stays up-to-date.   And with no limit to stream depth, teams can perform prototyping efforts in parallel and/or perform prototyping based on existing prototypes by adding another child stream!  Furthermore, the pattern works for any size development activity or team, even for us Team-of-One developers with tons of ideas, fast fingers, and a few green screen x-terms (2 space, 80-char wrapping of course – <chuckle>).

This is a perfect example of how AccuRev empowers the developer to take control of their own development.  Creating streams is extremely easy and the stream browser provides unprecedented visibility into the entire development process.  With the right amount of security in place (Locks, ACLs, Triggers), the critical release streams (left side of the stream structure) can be locked down by the CM Admins, but the development related streams (right side) can be fair game for developers to create an environment that suits their purpose, such as prototyping.

Does anyone have a good story to tell about how this pattern (or equivalent) helped with a major refactoring effort or library upgrade?

/happy prototyping/ – dave

by Chris Boran

Several years ago I happened to be browsing in my favourite local bookstore and one book in particular caught my eye: Martin Fowler’s Refactoring: Improving the Design of Existing Code

This is the book that changed my whole career. Up until that point, I had lived in a constant state of fear of change. I viewed old code as a house of cards – if I wasn’t very careful, it was all going to come down around me. Whenever my boss asked me for advice about what to do in a given area of the code, my answers were almost always similar to:

The risk of doing anything but the smallest possible change is huge – so either we do something that is an ugly hack that we will regret later, or we need to take the whole thing apart and re-implement the whole subsystem from the ground up.

And predictably, my boss’s answer was always something like:

Okay, we will live with the hack for now, but in the next release we will make time to do it right.

Of course every release we were forced to make many similar decisions. When the next release would come, the newly conceived product features would get priority over fixing code that was already working passably. In practice we might get lucky and be able to spend lots of time rewriting a single small subsystem, but introduce ugly hacks that would put 4 other systems on the map for future re-implementation.

The end result is affectionately referred to as a Big Ball of Mud. Yup, it is every bit as pleasant as it sounds. Life is just so miserable when you come to work every day knowing you are going to have to pack another layer of mud on the ball. You gripe about it. Your teammates gripe about it. Your boss gripes about it. Somehow you never seem to make a whole lot of forward progress.

As I read the book, I was at first skeptical. I thought that bad old code needed to be thrown away and not revitalized! But I wanted to see if it would work, and so I set out to try it. I was very careful to follow the techniques exactly as they are laid out in the book. I made sure not to use the word refactor when I really meant rewrite. I was careful to refactor the code every time I saw something I wanted to change and not just note it for later. I made sure that my refactorings were small. I took my time with it.

Obviously working this way required that I also be doing very good unit testing, but I had already bought into Test Driven Development. I was already writing unit tests for code before fixing bugs in an attempt to prevent regressions. Running these tests after each refactoring was not a big challenge for me.

Bit by bit I discovered the truth. By applying the refactoring techniques, I could take pieces that I thought needed to completely rewriten and make them better while I was fixing bugs in that area. I could kill two birds with one stone.

Then I discovered Eclipse. The built-in refactoring browser captivated me. Suddenly there was a good, fool proof way to do many of the common refactorings, and automation to keep them introducing new errors. My commitment to refactoring was completed and the defect rates in the code that I was responsible for maintaining declined dramatically. I was a convert. Since that time refactoring has been a cornerstone technique in my arsenal. I no longer lived in fear and loathing of old code!

One refactoring that I have still found to be painful, despite Eclipse’s facilities, is renaming files/classes. In most software configuration management (SCM) tools, renaming files can have unfortunate unintended side effects because the identity of a file is its path. This leads to a great many developers to name a class once, and then never change its name – even if that name does not make sense and the design of the code and the classes primary purpose and responsibilities change.

Luckily for me, I am a long time Accurev user. In Accurev, files are not identified by their pathnames, but rather by a unique id. This makes it possible to quickly and easily rename files with no negative side effect. However this process was inconvenient – I would have to drop out of Eclipse, rename the file with Accurev’s tools, and then refresh my workspace. Not ideal, but it worked. That is why I was so pleased when the Accurev-Eclipse Plugin was released – it integrated the Eclipse Refactoring browser’s rename actions with Accurev’s capabilities to make the whole experience seemless. Accurev has helped me to maintain well thought out, easy to understand designs despite constant evolution to those designs.

For those tracking the movement of luminaries such as Martin Fowler and Kent Beck or looking for scalability advice from the architects at companies like Orbitz, Ebay, and Linked-In… QCon ’07 in downtown San Francisco is the place to be!

The conference is packed with senior architects, software engineers, and open-source contributors galore — over 400 were rumored to be in attendance. With speaker topics ranging from enterprise scalability to Agile practices, the audience was nothing short of being at the top of their game. Huddled together at the entrance to the conference rooms were a number of vendors showing off new warez including AccuRev. Here’s a shot of Damon Poole & Cliff Utstein (top-right), Dave Thomas (left), and John Wall (bottom-right). The AccuRev booth had a constant flow of folks amazed at how the stream-based architecture brings a refreshing approach to managing software configurations and supporting agile practices. We also had some cameo appearances from existing customers like Authorize.Net and Orbitz.com.

Agile development methodologies is a major theme of the conference. For someone looking for advice on agile, just standing in the middle of the exhibit hall is all it takes — everyone is talking about best practices, success stories, and failed attempts.

We had a constant stream of people intrigued by our stream-based architecture and inherent support for agile practices. Here’s a list of common discussion points:

private workspaces: commit-early, commit-often

stream inheritance: merge-early, merge-often and sharing iterations early and automatically with parallel development efforts

issue tracking integration: assign, deliver and track development activity to stories/issues/features

continuous integration: integrations with cruise control, finalbuilder, electric-cloud, and others

refactoring: IDE integrations with eclipse, Intelli-J, Visual Studio support application-wide refactoring with version control

staged workflows: organize distributed teams and isolate integration areas from testing areas for explicit and repeatable access to known configurations.

snapshots: guaranteed reproducibility of labeled configurations for builds known to be ‘good’

reparenting: retarget active development to known good configurations for testing or stable development

If you didn’t have a chance to attend this years QCon, be sure to put next years event on your calendar!

/happy conferencing/ – dave

One of the greatest stories ever told is your software configuration management or CM process through the lense of a StreamBrowser. No other tool gives you the power to both visualize and manipulate your software development process in real-time. Over time, you’ll notice your stream hierarchy evolving into an elaborate tree effectively telling the story.

There are a number of techniques to help navigate the tree such as the GUI zoom feature and good naming conventions. But the unsung hero of stream organization is the pass-through stream. You probably already know about other types of streams such as dynamic, workspace, snapshot, and timebasis. Just like its siblings, the pass-through stream is also a configuration of source code but comes with a unique feature — all elements promoted to it are are automatically promoted to the pass-through’s parent stream, hence the name.

So what exactly are the benefits?

• Group logically related child streams for organizational purposes.
• A handle to reparent groups of streams without requiring an unnecessary promotion (to it).
• Set include/exclude rules to be applied to all children streams without the unecessary promote step.
• As an advanced technique, you can create a reftree on a pass-through to do software builds wherever the passthrough is currently parented — say last weeks build or last nights build — offering easy, visual, manual control over which configuration is built.

Lets put this knowledge to use. In the following diagram, you’ll see how a relatively sophisticated process model is cleanly organized with pass-through streams. There are a few pattern motifs used for organizing active development, nightly builds, system test builds, official release codelines, and customer one-offs. Notice how collapsing them all (left image) leaves the high-level process-centric streams nice and clean looking. Furthermore, notice how the names of the pass-through streams help tell the story of what activity is happening where.

Using pass-through streams is a passive refactoring technique that does not interfere with inheritance and can be introduced safely without disrupting development activity. In my experience, the more organized your stream structure, the happier the end users.

/happy refactoring/ – dave