Agile big design up front

I've been thinking about making tea and coffee for the past few hours. Or rather, I've been thinking about making project plans for how to make tea and coffee as an exercise to set the students in the Systems Analysis & Design course I'm going to be teaching from next week.

I've always liked using the process as a metaphor for what we have to deal with when writing software. If you want your Granny to understand what you do start off by writing a simple list of instructions for making a mug of tea or coffee. She'll wonder why you get paid for such an easy job. Now ask her what happens if she's run out of tea bags, coffee, milk or sugar? What if the power or water is off? If the kettle is broken? No clean mugs will mean adding a mug washing feature (and associated plan and error handling).

Once you've gone through it you'll have five lines of instructions of what to do normally and anything up to five pages (maybe more if you really go to town) of instructions of what to do in the case of various failures and how to take corrective action (and how to carry out those actions, and how to recover from failures in the recovery actions and so on and so on).

This is exactly of course what happens in our code. For the sorts of business systems that I work on there's typically a ratio of around ten-to-one of error detection and recovery code compared to code for dealing with the task as it is most commonly executed.

But that isn't really the purpose of this post.

Doing a big project plan like this up front is what the traditional waterfall process looks like. Do your planning all up front, think of as many contingencies as you can and work out what to do about them all. More agile tea makers just head to the kitchen and sort out the problems when (and if) they hit them.

Did any of you reading that last paragraph get confused about the switch between looking at making tea as a metaphor for a computer program and using it as a metaphor for the development process itself? I would be very surprised if you did.

After all, couldn't we write a project plan using a waterfall like process? Where somebody categorises the parameters and the relevant failure modes before delegating the task of writing the detailed instructions to others? We could also do it in an agile way, where the team has a ten minute meeting at the start of each day discussing ideas and problems and then they go away and write up and work on different aspects of it.

Go and (re-)read Joel's essay on the new Evidence Based Scheduling feature of Fogbugz. The process that he's describing sounds about as waterfall as it comes, and if it isn't quite a big design up front, it is certainly at least medium sized.

The question that he glosses over is how the project plan gets done in the first place. Now Joel doesn't normally have nice things to say about agile development processes, but I suspect (quite strongly) that the process that Fogbugz uses to create that project plan has more in common with the agile write-a-tea-making-project-planners than it does with the waterfall ones.

Even for the waterfall tea-making-project-planners somebody has to come up with the parameters they are going to use. And even if that is just one person it'll be done in an agile, iterative manner because we have start somewhere — it doesn't spring ex nihilo from whole cloth. If we go far enough back we will find something that has all the feel of agile and iterative processes.

Joel may prefer to write his software with an up front plan, but he still has to write the plan. And even if he has a plan for writing plans, somewhere it will go agile and iterative on him — it simply has to. He cannot decide up front how long it will take his developers to understand the features he wants written — it will take them as long as it takes them. His only choice is whether he forces them to do it before they write any code (waterfall), or whilst they're writing the code (agile).

Now for the real point of this post.

What happens when we move forwards in the process? The final artefact of any software development project is some software (well, duh!). If we look from the point of view of the software's users then we see that the software we deliver them is just one, huge, big, design up front. It's the biggest design up front we can have because we've already decided everything that the users of the software can do in the most excruciating detail possible — working lines of actual executable code.

You see, at the start of every software development project is an agile process, and at the end is a big design up front. Your choice as a development team is only how late in the project you want to stay agile, and how early you want to see what that big design is really going to look like.