Do you ever feel like things are out of control on your project, that you are adrift in a sea of conflicting priorities and requests? Do you suddenly find out at the last minute that you are the bottleneck and everybody is breathing down your neck asking you what is taking you so long to create the moveStuff() method but you had no idea that anybody even cared about moveStuff() or that you owned it? Do you ever find yourself in the exact opposite position, wondering why Sue and Bob didn’t get their stuff done that you need and then your boss walks by while you are surfing the net waiting for Sue and Bob? And who is Bob anyway?

The solution is simple! All you need to do is get everybody to move to Project. Well, if you have somebody you can spare full-time to keep Project up to date of course. Oh and I almost forgot, you’ll need to start using a requirements tool. But that’s it really, other than integrating them all together over the weekend and of course that’s assuming you’ve already gotten a CRM tool for workflow.

There is a simpler solution. It isn’t perfect, and it doesn’t solve all problems, but it can definitely provide the following benefits:

• reduce the chaos
• increase your vision into where you are and what’s going on
• reduce the number of status and/or project management meetings
• reduce the need to provide the same information over and over again
• simplify collaboration both locally and for distributed teams
• provide a more Agile workflow

The answer is to reduce the amount of rework that you are already doing. Right now you are probably storing defects in a defect tracking system, enhancements (aka RFEs, requirements, etc) in a requirements management tool (usually Excel or Word but sometimes an actual RM tool), and if you are using a project management tool it is probably MS-Project. All three of these product areas evolved to provide different aspects of project management for different groups of people and as a result they have lots of overlap. Considering how hard it is to coordinate three different systems, why not consider standardizing on one system for most of the work? The only question is, which system?

If we are going to try to do most of our project management work in a single tool, we should first decide what the interesting activities are. I believe they are: recording enhancement requests and defects as they are gathered by marketing or reported by users, load balancing, estimated completion calculation, critical path determination, work assignment, workflow, and reporting.

First let’s consider how well suited Project is for doing most or all of these tasks. Project is good at taking a small static collection of large tasks and doing load balancing, estimated completion, and critical path determination. Thus, it is mostly used for the very narrow task of project management of enhancements.

Next let’s consider requirements management. For whatever reason, most people use Excel or Word as their requirements management tool instead of a “real” requirements management tool. Excel and Word are just not appropriate for project management.

Lastly, there is defect tracking. A defect tracking system covers the assignment, tracking, workflow and reporting of defects. There is usually a higher volume of defects than enhancements, and they are usually smaller in scope and have a more complicated and often more time critical workflow. If it works well for defects, it should work equally well for enhancements.

Based on this analysis, it makes sense to extend the project management that you are already doing with a defect tracking system to include enhancements. A generic name for something that is either a requirement, enhancement, or defect is “work item.” By using work items to track all work, it is easy to see where you are and what remains to be done. Now you can use a similar workflow for enhancements as you do for defects, for instance from newly entered, to triaging, to assignment, to test development, to completion, to test, to integration, to delivery. You can easily run a query to see which work items have their code written but do not yet have any tests. Similarly, you can see which work items are done from a coding perspective and have tests but have not yet been verified as done by QA. This will give you a much more complete view of your overall project status and progress.

Whatever you are currently using for defect tracking it will be straightforward to start getting the benefits of managing defects and enhancements together. Just add a field that indicates if a work item is a defect or an enhancement. You may need to make a few more changes to accommodate a slightly different workflow for enhancements than you have for defects, but I think you’ll find it is worth the effort. For one example of how this can work, you can take a look at how AccuRev does it using AccuWorkflow.

In the prior blog on this topic I talked about the types of data I wanted to collect for developer reporting, and the basics I had to do to get the information out of the AccuRev software configuration management tool and into the Eclipse BIRT tool in Eclipse.

I promised charts, but before I get there, I realized I had to do a little more work to massage the data for use with BIRT. When you do an issue query in AccuRev (see prior post) the details for AccuRev users are only exported as the id number. To get this to look nice I ran a second command (accurev show -fx users) to get the user information as xml data. Most AccuRev commands have xml output in the format of <AcResponse>…</AcResponse> so I then ran this result through another xsl parser. My xsl file is straightforward and looks like this:

<?xml version=’1.0′ encoding=’utf-8′ ?>
<xsl:stylesheet xmlns:xsl=”http://www.w3.org/1999/XSL/Transform” version=”1.0″>
<xsl:output method=”xml”/>

<xsl:template match=”/”>
<users>
<xsl:apply-templates select=”/AcResponse”/>
</users>
</xsl:template>

<xsl:template match=”AcResponse”>
<xsl:apply-templates/>
</xsl:template>

<xsl:template match=”Element”>
<user>
<number>
<xsl:value-of select=”@Number”/>
</number>
<name>
<xsl:value-of select=”@Name”/>
</name>
</user>
</xsl:template>
</xsl:stylesheet>

This generated user tags which I could then use as a lookup when converting the issue data. The output of running this gave me the following xml file (names removed):

<?xml version=”1.0″ encoding=”UTF-8″?><users>
<user><number>1</number><name>…</name></user>
<user><number>2</number><name>…</name></user>
</users>

I could now run the issue xml through another xsl translation, which would use the xml above as a lookup table for user details. The issue xsl I used looks like this:

<?xml version=’1.0′ encoding=’utf-8′ ?>
<xsl:stylesheet xmlns:xsl=”http://www.w3.org/1999/XSL/Transform” version=”1.0″>
<xsl:output method=”xml”/>

<!– Begin Lookup table of users –>
<!– Setup key –>
<xsl:key name=”user-lookup” match=”user” use=”number”/>
<!– Setup variable for users lookup file –>
<xsl:variable name=”users-top” select=”document(‘activeUsers.out’)/users”/>

<!– Setup match for lookup of users –>
<xsl:template match=”users”>
<xsl:param name=”curr-user”/>
<xsl:value-of select=”key(‘user-lookup’, $curr-user)/name”/>
</xsl:template>
<!– End setup match for lookup of users –>
<!– End Lookup table of users –>

<!– Setup match for submittedBy –>
<xsl:template match=”submittedBy”>
<xsl:apply-templates select=”$users-top”>
<xsl:with-param name=”curr-user” select=”.”/>
</xsl:apply-templates>
</xsl:template>
<!– End match for submittedBy –>

<!– Setup match for assignedTo –>
<xsl:template match=”assignedTo”>
<xsl:apply-templates select=”$users-top”>
<xsl:with-param name=”curr-user” select=”.”/>
</xsl:apply-templates>
</xsl:template>
<!– End match for assignedTo –>

<!– Setup match for closedBy –>
<xsl:template match=”closedBy”>
<xsl:apply-templates select=”$users-top”>
<xsl:with-param name=”curr-user” select=”.”/>
</xsl:apply-templates>
</xsl:template>
<!– End match for closedBy –>

<xsl:template match=”/”>
<xsl:apply-templates select=”/acResponse”/>
</xsl:template>

<xsl:template match=”issues”>
<issues>
<xsl:apply-templates/>
</issues>
</xsl:template>

<xsl:template match=”issue”>
<issue>
<issueNum>
<xsl:value-of select=”issueNum/.”/>
</issueNum>
<status>
<xsl:apply-templates select=”status”/>
</status>
<state>
<xsl:apply-templates select=”state”/>
</state>
<severity>
<xsl:apply-templates select=”severity”/>
</severity>
<priority>
<xsl:value-of select=”priority/.”/>
</priority>
<dateSubmitted>
<xsl:value-of select=”dateSubmitted/.”/>
</dateSubmitted>
<dateClosed>
<xsl:value-of select=”dateClosed/.”/>
</dateClosed>
<dateCompleted>
<xsl:value-of select=”dateCompleted/.”/>
</dateCompleted>
<dateAssigned>
<xsl:value-of select=”dateAssigned/.”/>
</dateAssigned>
<submittedBy>
<xsl:apply-templates select=”submittedBy”/>
</submittedBy>
<assignedTo>
<xsl:apply-templates select=”assignedTo”/>
</assignedTo>
<closedBy>
<xsl:apply-templates select=”closedBy”/>
</closedBy>
</issue>
<!– xsl:apply-templates/ –>
</xsl:template>

<xsl:template match=”status”>
<xsl:attribute name=”name”>
<xsl:value-of select=”.”/>
</xsl:attribute>
<xsl:attribute name=”value”>
<xsl:choose>
<xsl:when test=”.=’New’”>
<xsl:value-of select=”’1′”/>
</xsl:when>
<xsl:when test=”.=’Reviewed’”>
<xsl:value-of select=”’2′”/>
</xsl:when>
<xsl:when test=”.=’Scheduled’”>
<xsl:value-of select=”’3′”/>
</xsl:when>
<xsl:when test=”.=’Closed’”>
<xsl:value-of select=”’4′”/>
</xsl:when>
<xsl:otherwise>
<xsl:value-of select=”’0′”/>
</xsl:otherwise>
</xsl:choose>
</xsl:attribute>
</xsl:template>

<xsl:template match=”state”>
<xsl:attribute name=”name”>
<xsl:value-of select=”.”/>
</xsl:attribute>
<xsl:attribute name=”value”>
<xsl:choose>
<xsl:when test=”.=’Verified’”>
<xsl:value-of select=”’3′”/>
</xsl:when>
<xsl:when test=”.=’Complete’”>
<xsl:value-of select=”’4′”/>
</xsl:when>
<xsl:otherwise>
<xsl:value-of select=”’0′”/>
</xsl:otherwise>
</xsl:choose>
</xsl:attribute>
</xsl:template>

<xsl:template match=”severity”>
<xsl:attribute name=”name”>
<xsl:value-of select=”.”/>
</xsl:attribute>
<xsl:attribute name=”value”>
<xsl:choose>
<xsl:when test=”.=’A'”>
<xsl:value-of select=”’1′”/>
</xsl:when>
<xsl:when test=”.=’B'”>
<xsl:value-of select=”’2′”/>
</xsl:when>
<xsl:when test=”.=’C'”>
<xsl:value-of select=”’3′”/>
</xsl:when>
<xsl:when test=”.=’D'”>
<xsl:value-of select=”’4′”/>
</xsl:when>
<xsl:when test=”.=’E'”>
<xsl:value-of select=”’5′”/>
</xsl:when>
<xsl:otherwise>
<xsl:value-of select=”’0′”/>
</xsl:otherwise>
</xsl:choose>
</xsl:attribute>
</xsl:template>
</xsl:stylesheet>

I had to scratch my head how to setup a lookup xml file, until I found this article on the IBM developerWorks site. It took some experimenting, but the xsl file above did the trick. I could have done a similar lookup for the status/state changes that are also in this xsl file, but it was quicker to enter these values directly. The side effect to keep in mind is because I don’t dynamically(today) get these values then if they change the xsl file needs to change.

After using this xsl file I get my new issue xml which looks like this (names removed and formatted for display):

<?xml version=”1.0″ encoding=”UTF-8″?>
<issues>
<issue>
<issueNum>1092</issueNum>
<status name=”Closed” value=”4″/>
<state name=”Verified” value=”3″/>
<severity name=”D” value=”4″/>
<priority>0</priority>
<dateSubmitted>943077600</dateSubmitted>
<dateClosed>1188926921</dateClosed>
<dateCompleted>1172186181</dateCompleted>
<dateAssigned>943077600</dateAssigned>
<submittedBy>…</submittedBy>
<assignedTo>…</assignedTo>
<closedBy>…</closedBy>
</issue>
<issue>
<issueNum>2614</issueNum>
<status name=”Scheduled” value=”3″/>
<state name=”Complete” value=”4″/>
<severity name=”C” value=”3″/>
<priority>0</priority>
<dateSubmitted>1007048693</dateSubmitted>
<dateClosed>1149106741</dateClosed>
<dateCompleted>1189024691</dateCompleted>
<dateAssigned>1120772200</dateAssigned>
<submittedBy>…</submittedBy>
<assignedTo/>
<closedBy>…</closedBy>
</issue>
</issues>

Phew. That being done, I now had a fairly automated method for extracting project issues, lookup tables for extraneous information, and xsl files to massage the data before processing in BIRT. Now I could bring the data into BIRT and begin setting up charts.

I spoke to some extent in the previous blog about the calculated fields I used in BIRT. I’ve since added a few more. I added a computed column which combines status and state, to make it easier to create certain chart groupings. Because these are two string fields it was as simple as putting in row["status"]+”/”+row["state"] as the expression.

I then also added the days to complete an issue, using this expression: (row["dateCompleted"]-row["dateSubmitted"])/86400

I now had enough information to create the 7 charts I wanted. I’ll highlight two that worked well for my purposes.

For completed issues by developer, I used the ‘assignedTo’ row as the Category (X) series, turned on grouping and set the interval to 0 so that each developer would get their own line. The Value (Y) Series was then the ‘issueNum’ row. From this I got the following chart:

Not bad, and it served my purpose. I could get a quick overview of how the issues were distributed. Some developers may cringe at seeing this type of information, as they fear they are not doing as much as their coworkers, but those reviewing the information have to keep in mind that there are lots of reasons for the issue distribution that is seen. Developers may be assigned more difficult work or not be on the project full time, and these basic numbers will not reflect the extent of these external factors. If you find you cannot easily answer these questions, then you need to find a way to track these external factors and include them in your issue management system.

For Severity Distribution, I wanted to take a look at what the general types of issues were that were resolved in the release. This was pretty easy. I created a pie chart, using issueNum as the series and severity as the category (grouping as text and getting the count). Using this I quickly got this chart:

Not a bad chart. Our use of severity A is in indication of crashes, so while some were fixed hopefully that is not the focus of the release. By the same token severity D and E cover a lot of cosmetic issues, so this release was a good distribution of the type of issues that should be addressed when preparing a release.

That’s about it for now. We’ve started adopting more agile practices, and in particular scrum methodologies, so now I want to revisit the BIRT configuration and start modifying it to generate burn down charts. Perhaps when I’ve made some progress here I’ll share some more.