Thursday, July 2, 2009

How would Agile practices help us meet deadlines more reliably?

We've been looking at adopting some Agile practices where I work. I've been doing a fair amount of reading on Agile in general and Scrum and Kanban in particular. Many thanks to some good people I follow on Twitter who have tweeted links to articles and blog posts I've found enlightening and educational.

In an excited but rather ad-hoc fashion I've been talking up all the good and interesting things I've run across to anyone who'll listen. One of my colleagues asked a great question yesterday along the lines of "so how exactly is applying Agile principles going to help us meet the customer's date?"

Try googling for that. I did. Unless things have changed since yesterday you won't turn up any really great articles that answer that very reasonable question directly. This forced me to try and articulate why I thought it would help. Below is my rather stream-of-consciousness reply to him. This could be all bollocks. This could be a gem. I'm a total neophyte here in the very early days of exploring and preparing to try and apply some of these ideas. But right now, this is where my thinking is. In the not too distant future I hope to be able to write from a position of experience. Will be interesting to contrast and compare then...

It has been proven time and again that you cannot reliably take a large set of vague needs and accurately estimate when they will all be implemented and tested. Waterfall methodology says if you fail at this it’s because you are advancing from one phase to the next prematurely, i.e. you needed to have done more requirements analysis and more design before moving to coding, testing etc. This can work if you have absolutely static requirements and a customer prepared to let you spend a hard-to-estimate amount of time doing the Big Up-Front Design™ phase without limit. Even then the phases that follow are still historically proven likely to slip unless significant padding is added to estimates and a hardline approach applied to scope creep (i.e. allow zero).

In most organizations true waterfall with the Big Up-Front Design™ is not feasible because:
  • organizations are not prepared to allow a large but unspecified period of time in order to get to a stage where you can predict the remaining 2/3 of the project time
  • requirements cannot be frozen for such long periods of time
  • the skills are not there to do requirements and design that are rigorous enough to allow a trouble-free implementation and testing phase

Therefore, if most organizations cannot successfully develop software in this way one should consider other approaches. The broad umbrella term “Agile” arose to cover a general approach with a number of specific “brands” of methodology recognized as adhering to Agile principles.

The well known ones are RUP, DSDM, XP, Scrum and Kanban. Many of the differences surround the extent to which a particular methodology is prescriptive. For example, XP specifies things you must have/do/generate that Scrum does not. Scrum specifies that you should have certain things, but fewer than XP. Kanban specifies very little. It is not uncommon to take what you decide are good practices from one methodology and incorporate them in another. For example paired-programming is a technique that originated in XP, isn’t mandated by Scrum or Kanban, but many people employing either of those techniques would still choose to use it.

The general themes that unify Agile thinking can be found expressed in the Agile Manifesto ( although they are pretty high level.

The concrete things that can be commonly identified as Agile practices include breaking work into fixed length iterations, estimating in story points, paired programming, holding retrospective meetings as a vehicle for continuous improvement, measuring team velocity, eroding barriers between different roles in the team, using daily stand ups to monitor progress etc.

So now to answer your original question, of how do these practices help us meet deadlines. There are two parts to this.

I believe we have to fundamentally alter what a deadline is. Instead of it being something we project after consuming some set of rough requirements, we want to set a fixed regular product release schedule.

Let’s say we deliver tracking every 6 months. Why? Because by doing this we now *know* already when we’re going to deliver it. The game now fundamentally changes from “I need you to get all this poorly specified vague stuff done by August” to the customer working with us to *get the highest value features added before the next release*

This is a much more favorable mindset to put them in. They become proper stakeholders and team members. They work with us to prioritize what they truly need over what they do not need. It’s Project Portfolio Management on a micro/individual project scale.

The notion that certain people in my organization advocate, that we can’t change deadlines or scope is not tenable. But they will continue to expect it in a mostly waterfall environment because it’s set up such that we get some needs, we estimate and then we are held to that. Yet it’s so obvious that this is impossible.

Therefore the game-changing approach of saying we release on a 6 monthly basis so now work with us to figure out what to get in there is I feel potentially very potent. Now sure, at the start of 6 months they/we will want to sketch out all the things they would like at the end of 6 months. But they have to be broken into stories and prioritized. And we have to accept that as we advance through the release cycle we *will* have to consider dropping some of the lower priority features. Why? Because we have fixed two of the three variables: budget (resources) and time (fixed date). Therefore you can only manage on that third axis.

If you can achieve buy in that the game changing alteration outlined in part one is the best known way to manage product development then you need a detailed way to operate during those 6 month release cycles.

Those techniques identified above will help. Specifically the really potent one is measuring velocity. Why? Because it will finally give us a way to know an approximation of team output. We have not had that before. If you have no measure of how much work the team can get done then projecting what can get done by when is somewhere between hard and impossible. This is why estimates are crap.

Using velocity involves the points based estimating and seeing how many points we can on average get through per iteration, known as velocity. Once you have that information and everyone has established a reasonably reliable pattern for generating points based estimates and burning through X number of points in an iteration you are able to do several useful things:
  • look at your backlog of prioritized features, tot up their value in points, and see when you can get them done by
  • figure out the impact of vacations more easily (25% of team on holiday? You can do 25% less points on average this iteration)
  • challenge the team to improve their velocity – typically via retrospectives

So the *$64-million dollar questions* is how do you get the customer to buy into this? Well I think the answer is to get them to recognize that the alternative doesn’t work and that people are employing Agile very successfully. One definition of insanity is to keep doing things the same way and expecting different results. Let’s not be insane. Let’s change they way we do things and be successful.