Posts Tagged ‘source code control’

« Older Entries
Newer Entries »

Continuous Integration: Building the Pyramid

March 13th, 2008 by jsherwood
Share|

I just saw the movie 10,000 B.C. (No, not the movie about when ClearCase was created). Certainly a diversion from everyday tasks, but I found myself focused on a couple of aspects of the movie. My first reaction seeing the workers moving the stones to build the pyramids (never mind the accuracy, just stay with me) was: how long does it take to move a single stone up that ramp? What if it took the group of workers pulling it a whole day? And what happens if they pull it all the way to the top and it cracks? Or is the wrong size? Now you’ve got to get rid of the stone and you lost a day.

Now take a look at the developers around you. You’ve labored hard all day, you feel pretty good about your accomplishments, take a rest and review it tomorrow. But you come in the next day and the build failed. You hauled that code to right place, only to find it cracked and the wrong size. What have you got now? You lost a day, your workers are all pointing the finger at each other and management wants to know why there’s a delay. You’re now spending the second day figuring out what was wrong on the first day, delaying the work you should have done.

Sounds pretty bad right? What you need is a way to continuously check what is going on. So you ask each developer to run a build, maybe even run some tests before promoting their code. Each developer seems to be on track, but now those integration points keep burning you. Bob only thinks in Windows, he keeps making references that work on his machine. When Julie, your UNIX guru, gets them they never build. Now you need some integration checking. But if Bob performs the integration on Windows he doesn’t have a problem (today) and Julie is sick and tired of managing Bob’s work. The build guru you hired would mutiny if she had to run builds all the time. You need some automation to relieve the pain.

And along comes Continuous Integration (CI). Martin Fowler wrote a great summary of CI (you can find the materials here). Besides this article there are many tools and even some books that are worthwhile for in depth knowledge of CI (take a look at Continuous Integration: Improving Software Quality and Reducing Risk – a 2008 Jolt Award winner). I’ll do them all a disservice and sum it up: frequently integrate autonomously.

I find the keyword here is autonomously. I don’t understand why developers spend so much time writing software for others to improve their processes, and yet they are willing to manually and repetitively do their own work. And to top it off they will rewrite software over and over. Just as you shouldn’t be spending your time writing a file i/o subsystem for the umpteenth time, you shouldn’t be running your build processes manually for the umpteenth time. If you can spend 2% of your time integrating builds instead of 15% of your time, isn’t that better?

Some time ago we had the problem I described above with Bob and Julie. We had Windows developers whose code would prevent builds on occasion with another OS. We have six primary OS that we ship, with many other variants. You can imagine the pain when nearing a release date, if you were to come into the office and find out you have no build. Once this happens it becomes a midday manual process to run and test the build. What drudgery. CruiseControl can solve this problem, and with a little work, can run with your custom build scripts. CruiseControl can greatly reduce the loss of nightly builds, as a matter of fact I’m hard pressed to think of the last time a nightly build failed for this reason, although we’re looking into commercial offerings that provide even more value. For now, having developers speak up (Hey Joe, that checkin you just did failed on Linux) can reduce your own personal stress level. You’ll have less hesitation to check in code just before you go home, with less need for connecting the dots because CI has your back. On occasion driving home my phone still buzzes, and when it does I know the build failed. Not the nightly build, CI alerted us soon enough so I still have time to fix it before that.

No worries. We’ll get that pyramid done on time.

No comments »

Posted in Best Practices

Tags:

Pattern for Continuous Builds

March 5th, 2008 by admin
Share|

Continuous integration (CI) is all the rage these days because merging, building, and testing (shared) configurations early-and-often is a good thing. Actually, it’s a great thing! After all, finding problems sooner rather than later benefits everyone. For some, CI means simply testing compilation. (Phew… it works. Ship it! haha). For those investing time in a full test harness, CI may mean frequently executing a suite of tests at various levels (unit, functional, system) to validate functionality and identify regressions. I’ve even seen other levels of CI to include lab testing, flight testing, or even customer acceptance testing for even the smallest of changes. Regardless of how you ‘do’ CI, I’ll show how I use AccuRev for continuous integration. [Keep in mind that this is one interpretation of the subject matter]

The Pattern. The stream-based nature of AccuRev makes it very natural to define separate areas for development, integration, testing, and release. Managing CI Builds with AccuRevAs seen in my example stream structure, I have an Integration stream as the first point of merging between individual project streams. This Integration stream is a great place to hook up a CI tool [Cruise Control, CC.NET, FinalBuilder, QuickBuild and perform nightly or per-promote builds. I prefer to create a snapshot before doing the build mainly because snapshot creation is atomic and their immutable configurations guarantee reproducibility. After creating the snapshot, I will pull the build from the snapshot name. You could build from the Integration stream directly (similar to the concept of a moving label), but creating snapshots makes it easy to visually identify with the build process and compare good builds from bad builds with simple stream diffs.  [Note: integrating any of the above mentioned CI tools is as simple as telling the build tool to pull code from a stream (by name) and then configure the build tool to execute at some frequency and notify people of the build status]

What about all those snapshots? At first, you may think, “Isn’t this going to create a gazillion snapshots? Won’t that take up a ton of (disk) space and totally clutter the stream browser view?” Well… No.

  • Snapshots are cheap. Snapshots are extremely cheap server-side entities consuming ~100bytes regardless of the number of elements they label… so go nuts! Snapshots mark transaction numbers, not elements! I say, always do what you need to solve important problems and answer tough questions even if that means creating a gazillion snapshots; just be sure to organize them.
  • Clean up as you go. Your CI build script (build.xml, Makefile, or doBuild.sh) can easily be instructed to remove a snapshot for every snapshot created. I’d recommend keeping around enough snapshots (say 3 to 10) to do valuable work such as comparing builds or serving as temporary baselines for developers who want to reparent. As you can see in the stream structure, AccuRev stores both active and inactive snapshots and it is easy to reactivate any snapshot if necessary (I’ve enabled the stream browser to show both; lower left corner option).
  • Group snapshots. I prefer to tuck logically related sets of snapshots behind a locked pass-through stream. The pass-through stream lets me collapse them all as a group and the lock prevents the pass-through stream from being accidentally being reparented.

Tip for very-long build/test cycles. Over the past few years I’ve encountered a few shops with single build/test cycles ranging from hours to days to complete. In this case, the concept of CI is slightly challenging because the notion of frequent builds is constrained. In this case, I’d recommend setting up two distinct test phases; quick and full. The “quick phase” is a quick pass sanity test only performing tasks such as compilation and unit testing — enough to let developers know they can continue on forward progress with little concern. The “full phase” is the full blown cycle, taking hours/days to complete, that completes all levels of testing such as compilation, unit testing, functional testing, system testing, etc. I would execute the quick phase early and often while the full phase may be once per week. As an additional step, I would mark the snapshot used for the full phase with a pass-through stream for the purpose of reporting configuration diffs or letting developers reparent their project streams/workspaces on the latest known good “certified” build.

Interested in continuous integration? Perhaps you’d also be interested in multistage continuous integration…

/happy building/ – dave

2 comments »

Posted in Patterns, Tips and Tricks

Tags:

Team of One Pattern

February 22nd, 2008 by dave
Share|

We all know that AccuRev is well suited for large enterprises with teams of developers spread across the globe. But what about the crack team of one developer? You know, those of use who go solo because we know we can do it faster, better, and leaner than any contrived dream team.

Can AccuRev really work for small teams including a team of one? … You bet your ASCII it does!

In fact, I use AccuRev for my own personal projects. They happen to be AccuRev integrations, but are software projects nonetheless [Vim Plugin, GNU Bash, etc]. At this point, critics may proclaim, “For one developer, you just need to commit to trunk and label for each release.” Well… that worked OK for the first and second release, but then came the need to maintain multiple versions in parallel with patch releases (due to wholescale refactoring per major release) as well as compatibility between corresponding versions of AccuRev (since these are plugins). The compatibility matrix completely obliterates any suggestion of linear branching/labeling… so fuh-get-abot-it. Time to graduate from the traditional branch-based tools.

Stream Structure for Team of One Development in AccuRev SCM

click to enlarge

The above picture shows my stream structure containing projects including vim-plugin and bash-completion. I’ll use the bash-completion project as a reference example to discuss my pattern of development. Even as a single developer, I found it critical to maintain a strict develop -> test -> release pattern simply because my development activities change day-to-day and typically turn into “It’s been a month since I looked at this code… time to roll up the sleeves and figure out what the heck I was thinking!” If I was forced to a single commit bucket (branch), I’d go nuts — trying to manage multiple patches, new development, updates to documentation, etc and then being forced to deliver it all because pulling out changes is about as fun as filling sandbags with a pitch fork… I’d rack my brains trying to keep it all straight especially since I have multiple projects going on concurrently!

My development process is as follows (annotated as steps 1-6 on the picture):

  1. Develop – After working in my private workspace on a unit of work for days, weeks, months, I promote to the “-test “stream. Then, continue working in the private workspace on the next set of work.
  2. Test – After unit testing and performing a clean-room functional test of all changes in my “-test” stream, I deem all changes “release ready” and promote to the top bash-completion stream.
  3. Release Candidate — The changes in the base stream represent a configuration that is good-enough to start a new codeline. I do NOT snapshot an official release X.Y (just yet). I first create a “dash-x” line to start the codeline (e.g. bash-completion-3.x for the 3.0, 3.1, 3.2, etc versions).
  4. Maintenance — In anticipation for even minor patch work, I proactively create a “-maint” stream to collect any upcoming maintenance changes based on the starting codeline. Initially, this stream will just be empty and identical in configuration to the parent ‘dash-x’ stream.
  5. Official Release — At this point, I’ve immediately created the”dash-x” and “-maint” streams in succession so they are all identical in contents – namely, all containing the next release. So NOW I create my first official “dot oh” release (e.g. bash-completion-3.0).
  6. Publish – With the official configuration under snapshot, I ‘pop’ the code, archive, and ship to my web server. La commedia e finite!

With the visual nature of the StreamBrowser and the ease of creating streams, managing multiple versions of multiple products with AccuRev is priceless. I use a simple, repeatable development pattern that lets me separate ongoing development work (workspaces) from upcoming changes being tested (-test stream) all separate from previous releases (dash-x) and patch development (-maint). And the best part about all of this is I can (and have, MANY times) come back to any project even months later and quickly ascertain the current state of the union — what’s in development, what’s in test, which releases are published, etc. Sweet.

Lastly, while I use this ‘dot oh’ pattern for my own projects, I even recommend it for large team development. It’s a great pattern and I hope you find it useful for your own stream management.

/happy releasing/ – dave

3 comments »

Posted in AccuRev, Best Practices, Patterns, version control

Tags: