Archive for May, 2008

« Older Entries

Right Process, Wrong Tool? Getting Ready for Agile

May 30th, 2008 by matthew d. laudato
Share|

Yesterday I was a panelist for a Webinar on agile tools, focusing on software configuration management (SCM), build and software process automation (SPA), the latter term referring to the set of defined, repeatable and measurable automated development workflows that engineers use to transform requirements into shippable software products. Contrary to what I’ve read about the disdain that some agile devotees have for tools, most of the attendees were hungry to know what features their SCM tool should have in order to support agile software development and SPA. Here are some of the highlights, and of course, my take on why I think AccuRev is the best tool for agile software process automation.

There are five key feature areas that an SCM tool needs to support in order to be ready for agile:

* Support for flexible process models

* Continuous integration support

* Support for issue-based development

* Efficient branch and code management

* Private version controlled developer workspaces

Let’s take a look at each of these in turn.

* Support for flexible process models. Agile is often one of several processes being employed within a software development organization. Unless your SCM tool is flexible and process-neutral, you will have a hard time implementing agile (say, for product development) and more traditional processes like waterfall (for example, for product maintenance work) in the same SCM tool. AccuRev streams are a natural way to model any process, and thus are a good fit when agile needs to coexist with other development processes. As for software process automation (SPA), AccuRev streams again are a great fit, since they enable users to model any arbitrary stages of code transformation that a development team sees fit to define as part of their process. By adding triggers and workflow to a stream hierarchy, teams can implement SPA directly in AccuRrev.

* Continuous integration support. Continuous integration is one of the core process elements associated with agile development. By building and testing frequently and acting on the results of tests, teams can uncover defects or test gaps earlier in their development cycle, saving time and money compared to such discoveries late in the cycle. But continuous integration goes beyond just testing the nightly build. With multi-stage continuous integration in AccuRev, code is automatically promoted up the stream hierarchy into more stable configurations as it passes tests. At each stage, continuous integration takes over to build and test, typically with a wider scope of testing as the code nears the release stage. Legacy SCM tools make this type of automated integration factory somewhere between difficult and impossible due to the complexity involved in setting up the hierarchy and in automatically moving and merging code as it flows up the hierarchy.

* Support for issue-based development. Apparently there is a lot of contention about the need for filing issues and defects in agile development. This has puzzled me greatly. While I’m in favor of developers identifying and fixing issues as they are discovered, you lose valuable process information when a defect or enhancement ticket is not filed and later associated with a code change. Without an issue that describes what the problem was, someone looking at the code changes for audit purposes or for group code reviews is at a disadvantage. Why was this code change made? Is it related to other changes? How long did it take? Was it done to fix a bug or to add a feature. In AccuRev, issues either in the integrated AccuWork issue tracking system, or in a 3rd party issue tracking system, can easily be associated with code changes via the AccuRev Change Package mechanism. This establishes basic traceability between issues and the code changes that developers make in order to satsify the requirements of those issues. Issue-based development is well-defined, repeatable and measurable – all hallmarks of good software process automation.

* Efficient branch and code management. Any time you’re working on more than one project, you need to isolate that project’s code from other projects. With agile and multistage continuous integration, even a single project requires multiple code lines in order to separate in-progress code from unit tested code from system tested code that is ready for release. If an SCM tool makes branching, merging and labeling difficult, teams tend to practice branch avoidance, which I sometimes like to call “fear of branching.” This is a classic example of letting a tool dictate what processes you can implement. In AccuRev, streams replace branches as the mechanism for isolating codelines. Since streams are represented inside of AccuRev as data separate from the actual file versions, creating streams is fast – really fast, like, a second or two – and managing a system with hundreds of streams spanning multiple projects and processes is easy.  For continuous integration, AccuRev snapshots and time-based streams are also fast and easy to create and manage, and give users a straight-forward way to “label” an interim or milestone codeline without having to place markers in thousands of source files.

* Private version controlled developer workspaces. Software developers are the heartbeat of any engineering organization. Executives at any development shop will tell you that hiring talented engineers and keeping them well-tooled and productive is the single largest challenge that they face. For agile, this is even more of a challenge, since coding cycles tend to be shorter, and thus anything that gets in the way of individual or team productivity tends to have a greater negative impact on the project. Private version controlled workspaces like the AccuRev workspace model improve productivity, since they enable developers to work in isolation (while they are ‘heads down’ coding). Private workspaces in AccuRev also give developers full SCM capabilites in their workspaces without the need to share in-progress code prematurely. By using the ‘keep’ operation, developers make safe copies of their work in the AccuRev repository, and later can ‘promote’ the code to an integration stream to combine their work with that of their teammates. Individuals are more productive in this way, and if continuous integration builds are frequently testing the integration stream, so are teams.

In a nutshell, agile requires tools, and these tools need to support different modes of operation than with other processes. SCM can help or hurt you in setting up and executing an agile process, so these guidelines are a way to help you get your SCM tool ready for agile - easy of course if your tool is already AccuRev!

If you’re interested, you can view the webinar recording.

Is Your Software Development Environment Agile-Ready?
Free On-Demand Webinar

4 comments »

Posted in Best Practices, SCM Resources

Tags:

Take Back Control: Using LDAP for SCM Authentication

May 15th, 2008 by matthew d. laudato
Share|

Take Back Control: Using LDAP for SCM AuthenticationYesterday, just for fun, I counted the number of times that I logged into a computer or website. Once to login to my PC. Once more to connect to the company network. Three times for the 3 different UNIX boxes I needed to work with. And (if you promise not to tell my boss) once to login to a popular online shopping site to cancel a book order that apparently was lost in shipping. That’s six times in a day – and a slow day at that.

All this logging in got me thinking about security, authentication and of course, software configuration management (SCM) systems. Most SCM users, unless they are in the computer security business or are otherwise paranoid, don’t think about what goes on when they type in their user name and password and press Enter. In this post, we’ll peel back the covers a bit and show how you can use LDAP as the authentication mechanism for the AccuRev SCM system.

Starting with version 4.6, AccuRev introduced the notion of a ‘custom’ authentication mechanism. If you boil it all down, there are only three things that you need to do in order to use LDAP authentication with AccuRev:

1. Tell the AccuRev server that you want to use custom authentication

2. Create users in AccuRev and in LDAP

3. Write a special AccuRev trigger that authenticates the users against an LDAP server

Let’s look at each of these in turn. First, a word of caution. As with any change to a shared production system, it is best to practice this in a safe environment. If you don’t have a spare AccuRev server laying around, you can always download the free 30 day, 5-user evaluation kit and use it to fine tune your new authentication process.

Now to the details. To tell the AccuRev server that you want to bypass the built-in authentication mechanism and use a custom method, execute the following command:

accurev authmethod custom

Next, you’ll need to create some AccuRev users. In this example, we’ll assume that you’re already using LDAP for other applications, and therefore user entries already exist in the LDAP server. A typical user in an LDAP server might look like this in LDIF format:

dn: cn=James T. Kirk,o=engineering,dc=enterprise,dc=com
objectclass: top
objectclass: person
objectclass: inetOrgPerson
objectclass: organizationalPerson
cn: James T. Kirk
sn: jtkirk
mail: jtkirk@enterprise.com
userPassword: jtkirk

Well, typical if they happen to be the captain of the most famous starship ever! But I digress.

In AccuRev, we need to decide how this user will be represented. In this example, we’ll use the LDAP ‘commonName’ attribute, which is shown above as ‘cn’, as the AccuRev username. Here’s the command we’ll use to create that user in AccuRev:

accurev mkuser “James T. Kirk”

At this point, we have a user represented in LDAP, and that same user represented in AccuRev. All we need to do is to tell the AccuRev server how to authenticate this user via LDAP. We do this via an AccuRev trigger known as the ‘server_auth_trig’. Here is some sample code for a server_auth_trig that does just that:

use Net::LDAP;
use Net::LDAP::Util qw(ldap_error_text);
use Net::LDAP::Constant qw(LDAP_SUCCESS
            LDAP_CONNECT_ERROR
         );

use XML::Simple;
use strict 'vars';

# Server info for contacting LDAP
my $LDAP_HOST = "localhost" ;
my $LDAP_PORT = "389" ;

# We explicitly list the subtree DN to use when rewriting the incoming username as an LDAP Bind DN.
my $ldap_baseDN = "o=engineering,dc=enterprise,dc=com" ;

# Default attribute name for binding.  This attribute is concatenated
# with the incoming username and the ldap_baseDN above to form
# a Bind DN.
my $ldap_bind_attribute = "cn" ;

sub main
{
    my ($xmlinput);
    my ($command, $ip );
    my ($username, $password );
    my ($result);

    # populate array using XML::Simple routine, reading from stdin
    $xmlinput = XMLin('-', forcearray => 1, suppressempty => '');

    # set variables
    $command = $$xmlinput{'command'}[0];
    $ip = $$xmlinput{'ip'}[0];
    $username = $$xmlinput{'username'}[0];
    $password = $$xmlinput{'password'}[0];

    # First, establish a connection to the LDAP server
    my $LDAP = Net::LDAP->new($LDAP_HOST, port => $LDAP_PORT) ;
    unless ($LDAP) {
                print "Unable to connect to LDAP server on host $LDAP_HOST at port $LDAP_PORT.\n" ;
                return LDAP_CONNECT_ERROR;
    }

    # Now that we are connected, rewrite the username as a DN and attempt to bind to the server
    my $ldap_bindDN = $ldap_bind_attribute . "=" .$username . "," . $ldap_baseDN;
    print "Attempting to bind as: $ldap_bindDN\n" ;

    my $mesg = $LDAP->bind($ldap_bindDN, password => $password) ;

    # 'code' method contains any error code from the bind call
    # including success, so we return it to the caller
    my $return_code = $mesg->code;
    print "LDAP_BIND returned: $return_code\n";

    # Now unbind to free the connection
    $LDAP->unbind;

    # return the auth code to the AccuRev server
    exit ($return_code);

}

# run main routine
&main();

The main trick is to ‘rewrite’ the incoming user name in the form of an LDAP Distinguished Name, or DN, and then to use that DN and the incoming password to bind to the LDAP server. Binding is a fancy word for logging into the server. Typically this is done by providing a DN (to uniquely identify the user) and credentials (in this case, a password).

As we said earlier, we’re using a convention in this example that the incoming user name represents the ‘commonName’, or cn, attribute of the user’s LDAP entry. We then construct a string by concatenating the cn with a hard-coded base DN, the latter representing the subtree within the LDAP server where the users exist. The resulting DN in this example is:

cn=James T. Kirk,o=engineering,dc=enterprise,dc=com

which is represented in the example as the perl variable $ldap_bindDN. If the bind is successful, the trigger returns a 0, and the user is logged into AccuRev. If the bind fails, the trigger returns a non-zero code, and the user login is rejected.

There you have it. A few simple steps and you can use the industry standard LDAP mechanism to provide authentication for your AccuRev users. LDAP is used in all sorts of enterprises, from education to technology companies to government, and so is AccuRev, so we’re glad to provide a way for our customers to use this powerful and ubiquitous authentication mechanism.

4 comments »

Posted in AccuRev, Tips and Tricks

Tags:

Continuous Integration: Methods of getting change

May 14th, 2008 by jsherwood
Share|

Continuous Integration: Methods of getting changeDo you remember the last time you were excited to go somewhere? Were you like a kid saying, “Are we there yet? Are we there yet?”

More likely you were the one getting ‘them’ there. I’m sure it got pretty annoying to keep hearing everyone in the car moaning, wanting some sort of distraction until they got there. Maybe you even had DVD players or some other distraction so you didn’t have to hear the questions.

Now think about how you use Continuous Integration (CI) . Do you have polling of your software configuration management (SCM) tool setup? What about your other process tools, do they poll your SCM as well? Guess what, it’s the same difficult trip. You have tools burdening your working SCM (who slaves away everyday to provide for these other tools!). Every hour/minute/seconds some tool is asking your SCM tool “Are we there yet?”. Doesn’t make a lot of sense does it? Think about how many tools you have running daily. You might have multiple CI machines setup, multiple reporting machines, deployment machines all asking the same question over and over.

Quite a burden.

Now think about what your users are doing during this time. They are looking for the same distractions you might have given the kids. They are off emailing their buddies about the latest game, or watching AccuRev on Youtube (OK, maybe something even more enjoyable on YouTube). Doesn’t sound very productive, yet these automation tools were meant to do just that, increase productivity.

So what do you do? Well, a lot of tools allow you to flip the model. Push your information, don’t pull it.

If you have a large enough development group this won’t be enough. Maybe there really are code changes going into the system every minute. If you also increase the granularity of the information going to your CI and reporting tools, these tools can then decide the correct time to be a burden.

You can also reduce the frequency that you give out the same information. If you have several tools (or stages in a tool) depending on the same answer, they can get the answer from a secondary source that gets populated once.

You also want to be diligent about your monitoring. If you see a periodic load on your tools, justify the load. If it looks strange that Kevin is checking the history of a stream every minute, it probably is. If instead you saw CIMonitor as the user it would be more explicit. And it would be obvious that this should change.

And really, changes like these apply to any tool you are using. Do you really need updated reports everytime a bug is fixed? What about every day instead? If you reduce the burden on your tools to a ‘necessary’ level, then they can be further used to answer other questions.

Are we there yet?

No comments »

Posted in Best Practices

Tags: