How would Agile practices help us meet deadlines more reliably?

We've been looking at adopting some Agile practices where I work. I've been doing a fair amount of reading on Agile in general and Scrum and Kanban in particular. Many thanks to some good people I follow on Twitter who have tweeted links to articles and blog posts I've found enlightening and educational.

In an excited but rather ad-hoc fashion I've been talking up all the good and interesting things I've run across to anyone who'll listen. One of my colleagues asked a great question yesterday along the lines of "so how exactly is applying Agile principles going to help us meet the customer's date?"

Try googling for that. I did. Unless things have changed since yesterday you won't turn up any really great articles that answer that very reasonable question directly. This forced me to try and articulate why I thought it would help. Below is my rather stream-of-consciousness reply to him. This could be all bollocks. This could be a gem. I'm a total neophyte here in the very early days of exploring and preparing to try and apply some of these ideas. But right now, this is where my thinking is. In the not too distant future I hope to be able to write from a position of experience. Will be interesting to contrast and compare then...


It has been proven time and again that you cannot reliably take a large set of vague needs and accurately estimate when they will all be implemented and tested. Waterfall methodology says if you fail at this it’s because you are advancing from one phase to the next prematurely, i.e. you needed to have done more requirements analysis and more design before moving to coding, testing etc. This can work if you have absolutely static requirements and a customer prepared to let you spend a hard-to-estimate amount of time doing the Big Up-Front Design™ phase without limit. Even then the phases that follow are still historically proven likely to slip unless significant padding is added to estimates and a hardline approach applied to scope creep (i.e. allow zero).

In most organizations true waterfall with the Big Up-Front Design™ is not feasible because:

• organizations are not prepared to allow a large but unspecified period of time in order to get to a stage where you can predict the remaining 2/3 of the project time
• requirements cannot be frozen for such long periods of time
• the skills are not there to do requirements and design that are rigorous enough to allow a trouble-free implementation and testing phase

Therefore, if most organizations cannot successfully develop software in this way one should consider other approaches. The broad umbrella term “Agile” arose to cover a general approach with a number of specific “brands” of methodology recognized as adhering to Agile principles.

The well known ones are RUP, DSDM, XP, Scrum and Kanban. Many of the differences surround the extent to which a particular methodology is prescriptive. For example, XP specifies things you must have/do/generate that Scrum does not. Scrum specifies that you should have certain things, but fewer than XP. Kanban specifies very little. It is not uncommon to take what you decide are good practices from one methodology and incorporate them in another. For example paired-programming is a technique that originated in XP, isn’t mandated by Scrum or Kanban, but many people employing either of those techniques would still choose to use it.

The general themes that unify Agile thinking can be found expressed in the Agile Manifesto (http://agilemanifesto.org/principles.html) although they are pretty high level.

The concrete things that can be commonly identified as Agile practices include breaking work into fixed length iterations, estimating in story points, paired programming, holding retrospective meetings as a vehicle for continuous improvement, measuring team velocity, eroding barriers between different roles in the team, using daily stand ups to monitor progress etc.

So now to answer your original question, of how do these practices help us meet deadlines. There are two parts to this.

PART ONE
I believe we have to fundamentally alter what a deadline is. Instead of it being something we project after consuming some set of rough requirements, we want to set a fixed regular product release schedule.

Let’s say we deliver tracking every 6 months. Why? Because by doing this we now *know* already when we’re going to deliver it. The game now fundamentally changes from “I need you to get all this poorly specified vague stuff done by August” to the customer working with us to *get the highest value features added before the next release*

This is a much more favorable mindset to put them in. They become proper stakeholders and team members. They work with us to prioritize what they truly need over what they do not need. It’s Project Portfolio Management on a micro/individual project scale.

The notion that certain people in my organization advocate, that we can’t change deadlines or scope is not tenable. But they will continue to expect it in a mostly waterfall environment because it’s set up such that we get some needs, we estimate and then we are held to that. Yet it’s so obvious that this is impossible.

Therefore the game-changing approach of saying we release on a 6 monthly basis so now work with us to figure out what to get in there is I feel potentially very potent. Now sure, at the start of 6 months they/we will want to sketch out all the things they would like at the end of 6 months. But they have to be broken into stories and prioritized. And we have to accept that as we advance through the release cycle we *will* have to consider dropping some of the lower priority features. Why? Because we have fixed two of the three variables: budget (resources) and time (fixed date). Therefore you can only manage on that third axis.


PART TWO
If you can achieve buy in that the game changing alteration outlined in part one is the best known way to manage product development then you need a detailed way to operate during those 6 month release cycles.

Those techniques identified above will help. Specifically the really potent one is measuring velocity. Why? Because it will finally give us a way to know an approximation of team output. We have not had that before. If you have no measure of how much work the team can get done then projecting what can get done by when is somewhere between hard and impossible. This is why estimates are crap.

Using velocity involves the points based estimating and seeing how many points we can on average get through per iteration, known as velocity. Once you have that information and everyone has established a reasonably reliable pattern for generating points based estimates and burning through X number of points in an iteration you are able to do several useful things:

• look at your backlog of prioritized features, tot up their value in points, and see when you can get them done by
• figure out the impact of vacations more easily (25% of team on holiday? You can do 25% less points on average this iteration)
• challenge the team to improve their velocity – typically via retrospectives

So the *$64-million dollar questions* is how do you get the customer to buy into this? Well I think the answer is to get them to recognize that the alternative doesn’t work and that people are employing Agile very successfully. One definition of insanity is to keep doing things the same way and expecting different results. Let’s not be insane. Let’s change they way we do things and be successful.

On matters of code style

(Or, “You know where you can shove your curly brace!”)

Source code is written documentation. It is the best, most accurate, in fact perhaps *only* accurate documentation you have. It completely and perfectly describes what your application does. No requirements specification, technical specification, UML diagram or anything else comes close to the comprehensive precision of your source code.

Now lots of written material is read for entertainment. The perhaps-soon-to-be-obsolete newspaper, novels, even for many people, non-fiction. But the first rule of software documentation is nobody reads it unless they have to. Except for a rare few, reading source code is not entertainment. When they are reading software documentation, people are reading it in pursuit of some goal. An emergency bugfix perhaps, or considering how to fit in a change to effect some new functionality. Perhaps looking to improve application performance or lift some existing code as the basis for another project.

Whatever it is, people have an objective when reading software documentation. And like I said, your source code is your *best* documentation. Unlike those written specifications and pretty pictures it is always up to date. It doesn’t suffer from the vagaries and ambiguities of the English language. It doesn’t require updating to reflect the latest design and behavior, it *is* the design and behavior.

Any non-trivial piece of software typically has quite a lengthy shelf life. It will often be far longer than perhaps initially anticipated, typically many years for business applications. Over its not inconsiderable life, with the ebb and flow of need bugs will be fixed, features added, code refactored, performance enhanced, third party libraries updated. All of this by a (more than likely) ever changing team.

Now most, if not all, source code is read on screen. Some people might like to print it out for scrutiny and reflection, but not the majority, whether seeking out a bug or peer reviewing a colleague’s work. Reading on screen is hard. Today I stumbled across a copy of Jakob Nielsen’s “Designing Web Usability” when rooting about in my furnace room. I dipped into it as I remembered him having something to say about content being read on screen. According to him people read 25% slower on screen, and in a study he conducted 79% of people scanned or skimmed rather than read word for word. I think it’s pretty fair to say that this is *exactly* how software engineers read code.

Consider then the trio of factors here:

1. source code is the best form of documentation you have for an application
2. it lasts a long time and will be read by a variety of people over it’s lifetime
3. it will be read on screen, likely skimmed and scanned by busy people on a mission

It seems blindingly clear to me then that the layout and style of your code is important. People need to be able to read, digest and work with source code as quickly as possible. I don’t think it’s any exaggeration to claim that many hundreds of hours of time and thousands of dollars can be saved over an application’s life if the code is easy to read. A simple back of the envelope calculation would demonstrate this for almost all business software.

So we need to make our code easy to read, need to ensure it has code layout and style to enable all those subsequent times it will be read, debugged, modified, refactored and reused. This of course begs the question, what characteristics make code easy to read? I believe it can be done by adhering to two simple principles:

1. write code that is easy to scan
2. use identifiers that are descriptive

What makes your code easy to scan is going to vary a bit; it's going to depend upon the language, but I think there are some general things to keep in mind. Here's my list, spawned for the most part from writing code for the last 7 years or so in Java:

• I want to be able to quickly see the start and end of classes. This is not normally an issue since there's one per file...unless you're using inner classes which can make things a little more exciting.
• I want to be able to to easily see significant independent pieces of code, specifically the start and end of methods, iterative blocks and conditionals
  
• Method arguments should be highly scan-able
  
• Judicious use of alignment and indentation can provide an easier to scan layout
• Remember, white space is free. Use it to improve readability.
  
• Any application of style and particular idioms should be used habitually. When you enter some code if you notice that it does not adhere to your preferred layout and style you should consider changing it. It's an easy and quick improvement that will probably pay dividends.
  

Descriptive identifiers need a bit of thinking. Sometimes the way something should be named is obvious. Sometimes however it's not, and merits discussion with peers. Sometimes a good name comes to you later after your first stab at things. This is where the "rename" refactoring in IDEs is really useful. Poor naming is worth fixing; left untouched people waste mental energy dealing with it that could be better spent. In particular give due consideration to:

• Good clear names for packages, classes, methods and variables
• Exploiting the existence of a domain specific vocabulary. However clarity is key; sometimes business users are inconsistent and you need to work hard to identify synonyms or subtle differences in seemingly interchangeable terms
• Brevity. Overly long unwieldy identifiers are not easily scannable. Similarly strive to reduce code. Less is more and all that...
• Avoiding redundancy in names, e.g. consider if you really need to pre-pend some system identifier to a series of classes, or if you have to append 'Servlet' to the end of all servlets that you write...
• Using plain, direct language. Overly abstract terms are often unhelpful. Complexity is inevitably there in most systems, but aim to simplify terms when reasonable to do so.

Finally, I was going to provide an illustrative example. Some code laid out in gnarly 'before' fashion followed by an 'after' rendering of almost poetic beauty. But since this post has languished in unpublished draft mode for months I decided I'm never going to get around to that. So you'll just have to use your imagination.

Chicken Makhani

Ingredients for stage one

• 1/4 cup vegetable oil
  
• 2 onions
  
• 1/2 cup chopped garlic
  
• 2 cups tomato sauce (not Heinz...the liquidy stuff in tins)
  
• 1 cup plain yoghurt (don't use low fat or fat free...it'll all go horribly wrong)
  
• 2 tbsp cumin/coriander (i.e. 1 tbsp each)
  
• 1/2 tsp garam masala
  
• 4 cloves
  
• 1 cinnamon stick
  
• 1" root ginger, peeled and chopped
  
• 3 bay leaves
  
• 2 cardomom pods
  

Ingredients for stage two

• 1 pint heavy cream
  
• 1/2 stick butter
  
• 1 tsp paprika
  
• 1/2 tsp ground black pepper
  
• 1.5 tsp salt
  
• 1/2 tsp hot chili powder

You will also need about 6 - 8 chicken breasts cut into bite size pieces.

Puree the onions, garlic and ginger in a blender. If you have a way to grind up the cinnamon stick and cloves do so. If not they can go in whole.

Heat oil till hot but not smoking, add cardomom and bay leaves. If you are not grinding up the cinnamon stick and cloves add them now. Stir for a few seconds then add the pureed mixture. Cook for 10 - 20 minutes stirring regularly to prevent it catching and burning. Add everything else from the stage one ingredients list and bring to a boil. The yoghurt may appear to curdle, but if it does don't worry -- everything will right itself shortly.

Now add stage two ingredients and return to a boil. Add the chicken pieces and simmer for 20 minutes or so until chicken is cooked and the sauce has taken on a slightly thicker consistency.

Serve with basmati rice and naan bread.

Naan Bread

• 1/2 cup plain yoghurt
  
• 1/2 cup milk
  
• 1/2 tsp baking powder
  
• 1 tsp white sugar
  
• 2 tsp dry yeast
  
• 1/2 tsp salt
  
• 1 tsp nigella (aka black onion seeds) 
  
• 2 tbsp melted butter
  
• 2 eggs, lightly beaten
  
• 3 cups plain flour
  

I've noticed that Indian restaurants in the US don't seem to put nigella in their naan bread whereas those in the UK do. Having spent most of my life in England I find them to be essential. I've made naan in the past without them though and they are OK, just missing that unique nigella flavor. In my opinion they are well worth seeking out. I found my last batch in an Indian grocery store. It used to be that Whole Foods sold them but last time I looked they have greatly reduced the variety of spices they now have. 

Start off by placing the yoghurt and milk in a large glass bowl and heating for approximately a minute or so in the microwave. Warming this up helps activate the yeast allegedly...

Next add the baking powder, sugar, melted butter, eggs and yeast to the bowl and stir well with balloon whisk to incorporate everything. Finally add the nigella seeds and stir to distribute.

Switch from the whisk to a wooden spoon and begin adding the flour. Add about a cup at a time and the stir in. When you have added all three cups if the dough is still too sticky add up to an extra half cup to make it easier to manage.

Turn out the dough and kneed for several minutes until it starts to have that nice sheen and elasticity that well kneeded dough has. Roll into a ball and cover with the glass bowl. Leave to rise for about an hour at room temperature. Dough should approximately double in size.

Once dough has risen divide it into at least four and as many eight pieces (depends how many you have to feed, how big you like your naan...)

Take each piece of dough and "encourage" it into the typical naan shape. This is a very approximate thing (for me at least) achieved through stretching, pressing and other general forms of manipulation as needed to get something reasonable looking.  In general the dough should end up pretty thin, a millimeter or two. No idea what fraction of an inch that is but not much...

Melt some more butter (perhaps 2 Tbsp or so) and brush each of your naan lightly on the top side.  Now place them butter-side down in a large hot skillet/frying pan that you have pre-heated and quickly brush the second side with a little butter. It should only take a couple of minutes before you need to flip the naan over. Use a fish slice to check on progress and ultimately turn the bread.

As each naan is finished transfer to a warm plate and keep covered with something (clean tea towel, foil, etc.)

These are great with any curry, but especially those with a lot of sauce.

Why we refactor code

Formulating an explanation in plain English for the non-technical

Late last week I received some “intelligence” that during a product meeting, one of our senior stakeholders exclaimed something along the lines of

“Why haven’t we made more progress on new features? Why have we spent all this time on refactoring? I don’t care about refactoring, it’s just fluff and no use to me.”

Unfortunately I missed this meeting. Or maybe it was fortunate. I don’t know that I’ve ever had to articulate to a non-technical person the purpose and benefits of refactoring code, and may not have done a great job if I’d tried to ad-lib. So over the weekend I mulled it over a bit, looking for a way to explain it that would resonate with someone like this.

The explanation had to be quick to explain and make patently clear the need for, and benefits of, refactoring. That itself got me thinking more deeply than ever before about why we refactor. Below is what I came up with. It’s all expressed in the context of the particular product that gave rise to this challenge, but I think the argument holds up across the board.

Here’s my basic premise: building software is like a game of Jenga™, and if you don’t do an appropriate amount of refactoring over its life the game will end sooner rather than later and everything will come crashing down. If that sounds odd, bear with me and I’ll explain. Let’s start with some background on the product that had given rise to the question:

• The product is a framework from which we build dozens of in-house web applications. They are all basically CRUD applications which vary a little from the functionality of the main framework (additional fields for example).
• The framework has had 7 releases over 5 years
• The first release had just over 1100 classes, the latest release has nearly 1800. That’s a 60% increase.
• Over its lifetime, at least 18 different developers in three different offices in two different countries have worked on it. There’s never been more than five people working on it concurrently so that’s a fair amount of staffing changes.
• The applications we build from this framework have the expectation that they can upgrade (or migrate as we call it) to newer releases of the framework than that which they were built upon

Now let’s build (no pun intended) the Jenga analogy. Consider each block in the stack is some piece of code, a class perhaps. Now obviously classes are more complex than this – they vary in size and have more complex dependencies, but as a simple model this will work.

Now, each release of the framework involves coding. That often means changing an existing class: pull out a block, "tweak" it, and then of course stack it on top. This represents how modifying existing code can lead to a more fragile situation. Unlike regular Jenga where you just use the existing blocks in the tower, coding involves adding new blocks. So your tower gets bigger over time, potentially making it a little more wobbly too. The other notable difference to help complete the analogy would be that whereas Jenga is played with 54 blocks, my framework here started with over 1100. And of course aggressive deadlines mean one doesn’t always have enough time to carefully consider how to position those blocks to best effect, but that’s how it goes...

After a few releases, our tower is looking pretty wobbly, although it still “works” just fine to the casual observer. After all, it’s still standing.

Business users: “We need you to add this new stuff.”

Engineers: “But those blocks are a completely different shape. Some of them are really oversized, and heavy. Besides, have you seen the foundations?”

Business users: “Yeah but if you could just tack them on somehow, that’d be great.”

See the problem is that the blocks are all rather precarious now. And some of them we never wrote ourselves we just use them. But they’re old, unsupported and clunky. We need to replace them or update them. We need to shore up the foundations. We’ve also noticed in the last five years that the business uses them in ways we didn’t appreciate when we first assembled them.

We need to reorganize the blocks a bit. We need to get rid of some of them, we need to simplify them. We need to shore up the foundations. We need to make some new blocks too, better suited to the way we have learned they actually get used. Considering how much we use them that’ll save money. In short, we need to refactor.

Stuffed Tomatoes

This is a recipe my wife and I discovered when toying with veganism (which was ultimately a failure - although we did pick up some good additional food.) Anyway, it was nice enough that it's been kept around and we still make it from time to time. The original recipe has you bake it in the oven, which is good...but to take it to a whole new level I highly recommend doing this over charcoal with some hickory wood chips for extra flavor. If you take the BBQ approach you don't want terribly hot coals, so either cook something else first or don't place the tomatoes over direct heat.

• 1/2 cup of orzo pasta (the small stuff that looks like rice)
• 4 large ripe tomatoes (such as beefsteak)
• 1/3 cup of pine nuts, briefly toasted in hot skillet
• 2 cloves of garlic, finely chopped
• 2 tbsp finely chopped flat leaf parsley
• 2 tbsp roughly chopped basil leaves

Cook the orzo according to instructions. Drain and rinse in cold water till chilled so it will keep without sticking together.

Whilst the orzo is cooking, slice the top off each of the tomatoes and then scoop out the insides using a spoon. Chop the tomato pulp and place it in a mixing bowl. Add the pine nuts, orzo, garlic, parsley and basil. Season with some salt and black pepper then mix well.

Stuff each of the tomatoes with the mixture, top with previously removed slice and drizzle with some olive oil.

If cooking in an oven, have it preheated to 375 degrees F, place on a foil covered baking sheet and bake for 20 minutes. If cooking on the BBQ, use your skill and judgment... :-)

Once cooked, serve with a generous amount of yellow pepper coulis which can be made as follows: fry one chopped onion and 3 chopped yellow bell peppers in a little olive oil. After 5 minutes add half a cup of water and some salt and pepper. Simmer until very soft, perhaps 15 - 20 minutes. Add more water if necessary. When done puree in a blender.

Gazpacho Soup

I really like this soup, the flavors are amazing. Truly one of those things where the whole is greater than the sum of the parts. Also, the fact that everything in it is raw (besides the bread) still strikes me as some kind of magic.

• About 2lbs very ripe tomatoes
• 1 large English-style cucumber, peeled and diced
• Half a baguette - a little stale is considered better. For those avoiding white flour I've used wholewheat bread too and it seems OK.
• 1 green bell pepper, de-seeded and and chopped reasonably small
• 1 cup (a little under half a UK pint) extra virgin olive oil
• 6 scallions (spring onions) roughly chopped
• 1 - 4 gloves garlic (how much do you like garlic?), roughly chopped
• 3 tbsp white wine vinegar
• 1 tsp paprika
• salt and pepper to taste (probably 1/2 - 1 tsp salt is what you want)
• 1 pint or less water

Tear the bread into small pieces and leave to soak in 1 cup of water. Immerse the tomatoes in just-boiled water briefly and then "encourage" the skins to split after a few seconds with a sharp knife. Pull them out of the water and remove skins. Chop tomatoes, removing seeds first.

Place chopped tomatoes into a large bowl along with the pepper, cucumber, scallions, garlic, bread and olive oil; stir to combine. Add the white wine vinegar, paprika, salt and pepper. Lastly add remaining water (1 cup). Stir again to fully combine.

Now transfer the contents of the bowl to a blender - you may need to do this in two or three batches due to the volume. Puree the mixture.

Obviously you can use more or less water depending on the consistency you prefer.

Once blended the soup should be chilled in the fridge for at least an hour. This also allows the flavors to really develop too.

To serve you can add a garnish of avocado cubes, diced red peppers, sprinkle of paprika etc.