Posts Tagged multistage continuous integration

Continuous Integration vs. Continuous Perfection

Continuous integration is about confronting your risk. It’s about giving rapid feedback and responding to problems that you may encounter in your development environment.

If you’re Agile, you know one principle is adaptation. It’s not about sticking to the plan; it’s about exposing and responding to change in your development cycle.  This is why the principles of continuous integration and Agile walk hand in hand.

Delaying the inevitable risk of code integration doesn’t mean the risk will disappear.  Each line of code you change is a risk. Each commit you do is a risk. You can’t avoid the risk by putting it off.  Just like anything, if you were going to bottle up all of that risk, let it ferment for a few months and open it later… I’m pretty sure you wouldn’t want to drink the whole bottle in one gulp. It really only takes one sip to know the concoction is bad.

Continuous Perfection is one of the reasons we bottle up the risk. Let me give you an example.

We set up continuous build on trunk, we encourage our developers to integrate early and often, life is good…. until the day of the broken build.

If it’s broken for even a day, panic ensues, we’ve issued blame to developers for something they should not be responsible for, a stoppage in work for organization. People will feel rushed to resolve issues, the QA team will stop testing, and our release engineer will struggle to roll back code. Essentially, people will be running around like chickens with their heads cut off.

Our management teams do what they do best, they listen to the release team, QA team and the dev team. “Joe Developer broke the build, it’s his fault that we can’t work today”.

When did the broken build become such a big deal?

This scenario may repeat itself, over the course of several weeks our mangers will react, and start asking developers to put in process to avoid breaking the build. Our teams will start to feel that their code must be perfect before they commit. This problem is called Continuous Perfection, it’s the idea that the code must be perfect before it’s committed to the repository.

And doesn’t this seem to work against the principles of continuous integration? The idea here was that we integrate early and often to avoid Integration Hell and the Big Bang merge. We want to confront and work through our problems right now. But now we’ve started to go back to what we do naturally. Isolate changes, put off the inevitable and work off of our private branches. Our developers do local builds to make sure the builds work correctly, and jump through hoops to avoid committing to early in fear of being blamed. We are back where we started, with a huge merge at the end of the project that could fail everything.  We bottled up all our risk, and ended up in a more dangerous place then we intended.

But what can we do to solve this problem ?  To be continued…

Posted in: AccuRev, Agile, Best Practices, Integrations, SCM Resources, Tips and Tricks, version control

Leave a Comment (1) →

Agile 2008 – Agile Skeptic?

By Josh Sherwood, AccuRev engineer

 Agile 2008   Agile Skeptic?

I had the opportunity to co-present a talk on Continuous Integration at the Agile 2008 conference this past week. While there I took the opportunity to attend a variety of sessions. Workshops discussing Business Value, Experience reports describing transitions from Waterfall to Agile, Aspects of Leadership in and Agile Environment and many others. And okay, I’ll admit that I have been a skeptic of a variety of the Agile Processes. Our CTO, Damon Poole, has been an Agile advocate for some time now. He and I have had ongoing discussions about the value of Agile practices. We would talk about the iterative model, going back and forth about the value and how to describe it to different groups. We’d eventually come to an understanding, but there were still many elements that I have been skeptical of and found difficult to see their value.

In addition to the elements of Agile practices that Damon and I would discuss, I started taking a look at other Agile topics. I looked at things like estimation, use of 3×5 cards and even pair programming. I didn’t see the value in the sticky note processes, I mean using a paper process to discuss a digital product? I didn’t see the value in pair programming, especially for consulting companies where your development hours translate directly into billable hours.

But then I came to the conference. I listened to people like Richard Sheridan talk about his experience with Pair Programming, why and how it worked for him. I listened to experience reports from companies like Healthwise, and heard about the difficulties they overcame to improve prior processes and move into Agile. And I spoke with Scrum trainers, who were encouraging their customers and helping them overcome the hurdles of understanding, while remaining flexible about sharing Agile ideas.

What was great about things like the Pair Programming was the team participation. Not only was Richard there talking about their factory model, team members were in the audience, talking about what they take away from the model. They talked about some of the things an individual developer would run into. You know those times, where you are working on a particular problem and just can’t quite figure out how to break the bottleneck. You wander around, get some coffee, or browse the web checking the Olympics. With their pair model, now that bottleneck has two developers working on it. No they’re both not checking their email, they are floating ideas back and forth, bringing to bare the problem and coming to some resolution. With a close proximity to other pairs they are also propogating these ideas farther out amongst the group. Yes there is training and not everybody can manage to work in pairs, but by using this model and actively rotating the pairs they ensure there are no knowledge silos. If someone is out one day there is more than one person who can tackle the problem at hand.

It has been a community of ideas this week. Some people use Scrum but don’t perform retrospectives. Some people model agile practices, but don’t follow a specific practice. Some people are taking the team based processes and developing large scale models for larger development tasks.

As I sit here building out the story for some of our future ideas, I’m encouraged that we can adopt more of the Agile practices than we have in the past, work through some of the confusion those ingrained in waterfall have with new processes, and further the practices that have allowed us to deliver innovation at a faster rate than we could with older models.

Posted in: Agile, Events

Leave a Comment (1) →

Pattern for Continuous Builds

Continuous integration (CI) is all the rage these days because merging, building, and testing (shared) configurations early-and-often is a good thing. Actually, it’s a great thing! After all, finding problems sooner rather than later benefits everyone. For some, CI means simply testing compilation. (Phew… it works. Ship it! haha). For those investing time in a full test harness, CI may mean frequently executing a suite of tests at various levels (unit, functional, system) to validate functionality and identify regressions. I’ve even seen other levels of CI to include lab testing, flight testing, or even customer acceptance testing for even the smallest of changes. Regardless of how you ‘do’ CI, I’ll show how I use AccuRev for continuous integration. [Keep in mind that this is one interpretation of the subject matter]

The Pattern. The stream-based nature of AccuRev makes it very natural to define separate areas for development, integration, testing, and release. Managing CI Builds with AccuRevAs seen in my example stream structure, I have an Integration stream as the first point of merging between individual project streams. This Integration stream is a great place to hook up a CI tool [CC.NET, FinalBuilder, QuickBuild and perform nightly or per-promote builds. I prefer to create a snapshot before doing the build mainly because snapshot creation is atomic and their immutable configurations guarantee reproducibility. After creating the snapshot, I will pull the build from the snapshot name. You could build from the Integration stream directly (similar to the concept of a moving label), but creating snapshots makes it easy to visually identify with the build process and compare good builds from bad builds with simple stream diffs.  [Note: integrating any of the above mentioned CI tools is as simple as telling the build tool to pull code from a stream (by name) and then configure the build tool to execute at some frequency and notify people of the build status]

What about all those snapshots? At first, you may think, “Isn’t this going to create a gazillion snapshots? Won’t that take up a ton of (disk) space and totally clutter the stream browser view?” Well… No.

  • Snapshots are cheap. Snapshots are extremely cheap server-side entities consuming ~100bytes regardless of the number of elements they label… so go nuts! Snapshots mark transaction numbers, not elements! I say, always do what you need to solve important problems and answer tough questions even if that means creating a gazillion snapshots; just be sure to organize them.
  • Clean up as you go. Your CI build script (build.xml, Makefile, or can easily be instructed to remove a snapshot for every snapshot created. I’d recommend keeping around enough snapshots (say 3 to 10) to do valuable work such as comparing builds or serving as temporary baselines for developers who want to reparent. As you can see in the stream structure, AccuRev stores both active and inactive snapshots and it is easy to reactivate any snapshot if necessary (I’ve enabled the stream browser to show both; lower left corner option).
  • Group snapshots. I prefer to tuck logically related sets of snapshots behind a locked pass-through stream. The pass-through stream lets me collapse them all as a group and the lock prevents the pass-through stream from being accidentally being reparented.

Tip for very-long build/test cycles. Over the past few years I’ve encountered a few shops with single build/test cycles ranging from hours to days to complete. In this case, the concept of CI is slightly challenging because the notion of frequent builds is constrained. In this case, I’d recommend setting up two distinct test phases; quick and full. The “quick phase” is a quick pass sanity test only performing tasks such as compilation and unit testing — enough to let developers know they can continue on forward progress with little concern. The “full phase” is the full blown cycle, taking hours/days to complete, that completes all levels of testing such as compilation, unit testing, functional testing, system testing, etc. I would execute the quick phase early and often while the full phase may be once per week. As an additional step, I would mark the snapshot used for the full phase with a pass-through stream for the purpose of reporting configuration diffs or letting developers reparent their project streams/workspaces on the latest known good “certified” build.

Interested in continuous integration? Perhaps you’d also be interested in multistage continuous integration

/happy building/ – dave

Posted in: Patterns, Tips and Tricks

Leave a Comment (2) →
Page 2 of 2 12