Category Archives: Agile

Speed by removal

If the goal is to get software out fast, a way to get there is to remove unnecessary work. Like this:

Contracts over the project scope. Plans change. Contracts either slow work down by necessiating negotiation before change, or speed it up by making change as easy as possible.

Deadlines. They seem like a good idea: by promising we will be ready by some day we commit ourselves to stripping away the unnecessary work. The problem is that committing to a deadline requires us to make guesses about the scope and implementation of the project before we have the required information. If our guess turns out optimistic, we miss the deadline. If it turns out pessimistic, we have unnecessarily postponed the delivery.

Coding. Many problems can be solved with a pen and a paper, or using existing software. Developing code is slow and expensive.

How we measure progress defines how we work

We want to be fast and reliable in our work, so we want to measure our progress. The question is what is the right scope for the measurement? When do we start the clock and when do we stop it? This is a fundamental question that in practice defines the way we work.

Let’s say the phases of our value chain — be they implicit or explicit — are like the picture on the right. To start with a narrow example, if we choose to measure our progress from the start of coding to the end of coding, that drives us towards very careful up-front planning and design of the system. We will want to be sure that when we start the coding, we will have as thorough understanding of the system as possible. We will be using something akin to waterfall as our development method.

On the other hand, if we measure from the start of the idea to the end of coding, we will end up with something like cowboy coding. We want to do as little design as possible, because that takes time from coding, which is the crucial part. We don’t need to care about testing, because that is out of our measurement. (Cowboy coders, almost by definition, don’t measure anything explicitly. But the argument goes in the other direction: if we measure things this way, we end up with something akin to cowboy coding.)

What measure leads you to Scrum? The methodology itself does not mandate where the boundaries are, but the way I have seen most Scrum teams implement it, is from the start of UI design or technical design to the end of testing or deployment. In theory, these kinds of Scrum teams strive to deliver working software but don’t care if it actually produces value or not. Product owner should take care of that part, although it is a bit more complicated than that in practice.

The things we measure set in practice the boundaries of our business. Even though we may focus on a certain part of the process—e.g. from IX design to deployment—if we choose to measure our success more broadly—e.g. from end to end—we will become invested in the result of the broad interpretation. This entails that we will in one way or another want to help our customers and suppliers in doing their part of the work.

So what is the right measure? There are two conflicting forces pushing us to different directions. What the business ultimately wants is quick creation of business value. But on the other hand, work is easier to organize and buy when it is modularized with clean interfaces. For example, if a company modularizes interaction design well enough, so that its input and output are clearly defined and of good quality, then it is easy to buy IX design from that company, and the measure of progress from the start of interaction design to the end of interaction design makes sense.

In my experience, success is often measured too narrowly, leading to many of the problems in our industry. The earlier in the process you start the clock the more adaptative the process will be. Likewise the later in the process you stop the clock the more relevant the feedback. Development of complicated software is not good enough yet to use modularized approach in its value chain. We need a more integrated approach.

Answer the Right Questions with Prototypes

Prototypes validate ideas. Rather than basing plans on guesses, you create something concrete and put it to the test. But what kind of a test? That depends on whether you are aiming for a sustaining or a disruptive innovation. The distinction helps us ask the right questions and build the right prototype. As a consequence we eliminate waste and create an effective innovation process.

I posted to Leonidas blog today: Answer the Right Questions with Prototypes.

Agile charted

Here’s how I think of agile.

Traditional software development has the problem that it gets feedback only after it has lost its ability to make relevant changes.

Agile on the other hand aims to get the feedback quickly while still having room to maneuver.

The upper right-hand corner of the chart is the sweet spot for development. It is the place where learning is possible and true innovation can occur. This is the heart of agile development. If your agile process leads you to the lower left side for whatever reason, you are doing it wrong.

I’ve pessimistically drawn both arrows’ ends’ in the lower right. From innovation point of view, the goal of course is to stay as long as possible on the upper right, and for example Facebook has kept in that general area for admirably long. But there seems to be a very real gravity that takes successful products down on the chart. This is The Innovator’s Dilemma: successful competition in an established market requires sustaining innovation, and companies that invest heavily in it tend to lose their ability to change direction. (Example: Nokia’s phones in 2000-2010.)

But while the individual products and product families are brought down by gravity, a company may escape it by cultivating disruptive innovation. Apple and 3M show that it is possible, but it appears to be very hard, since there are not many companies like that.

How to transform an organization

To me, the big theme of this year’s Scan-Agile conference was that transforming individual teams to agile is not enough – the entire organization needs to change. Converting a team to, say, Scrum is a waste of time if the increased output and flexibility cannot be used by the organization. It can be even harmful if it puts more pressure on other places of the organization. (And of course, it may be impossible for the team to change if the organization doesn’t.)

Based on the teachings of the conference, if I had to transform an organization, how would I do it? Here’s how:

  1. Can you draw the value stream map? If not, simplify your product offerings until you can.
  2. Draw the value stream map. Find out where the bottlenecks are.
  3. Slow down the value stream to the speed of the bottleneck.
  4. Find ways to speed up the stream at the bottleneck.
  5. Go to 2.

Any process changes, e.g. the introduction of agile development practices or beyond-budgeting principles, should have the goal of improving the value stream. If the change cannot be justified by looking at the value stream, it should not be made.

I don’t know much about value stream mapping or theory of constraints, but I think I ought to. I hear The Goal is the book to read.

I have only tinkered with process changes on the team level. You with more experience, please tell me, did I understand things correctly?

Let’s change hankintalaki

I’ve heard from many mouths that hankintalaki (the Finnish law governing publicly funded purchases) prevents effective software development for the Finnish public sector. The parliament is going to make changes to the law this autumn. Now would be a good time to tell our representatives what we think.

To start with, I would like to understand the exact problems with the law. How does it inhibit success of software projects? How should the law be changed?

Also welcome are any ideas about how to express our concerns effectively to lawmakers. I am trying to get the parliaments’s tietoyhteiskuntaryhmä (Information Society Group) hear one of us, but as far as I know, nothing else has been planned so far.

I also sent this question to the Agile Finland mailing list. Please let me know if you write about this on your blog. I’ll summarize the answers in a later post.

Learning and doing

Question

Think about your last project. How large portion of the effort was spent on writing actually useful features for the customer?

We are talking about actually useful here, from the customer’s point of view. Do not include the time spent on testing, meetings, waiting for the code to compile, duplicate code that could be refactored away, etc.

Do you have an answer? How much is it?

My answer

Let me do a ball-park estimate of a project from my past.

The codebase consisted of 80,000 lines of code. Roughly half of those were unit tests, developer scripts, and other non-production code so that brings the number down to 40,000 lines. At least half of the code could have been refactored away: 20,000 lines left. And about half of those lines were for features with negligible purpose, such as unnecessary user configuration options. So, I would estimate that 10,000 lines of code were actually useful.

Including the entire project team – developers, testers, project managers, customer representatives – the project took about 10 person-years. That is roughly 2,300 person-days, 17,000 person-hours, or 1,000,000 person-minutes.

Now, if we generously assume that each line of code takes 1 minute to type, it took about 10,000 minutes to type in the actually useful code. That is 1% of the 1,000,000 person-minute total.

So my answer is: about 1% of the time was spent on writing actually useful features for the customer.

Now, I know what you must be thinking: what a crappy project. But no, not at all. The project received high praise from the customer and spawned more project orders. Both the customer organization and the development organization started an effort to spread some of this project’s methods to other projects. This project succeeded.

Even in a successful software project only 1% of time is spent on writing actually useful features.

Learning cycle

The other 99% goes to finding out what are we trying to do and how we can do it.

Discovery cycle

We are learning about the product we are building, the interfaces we are using, the customer domain, and so on. We even learn about the code we wrote ourselves. Every bug report is feedback and every bug fix a start of another iteration on that minicycle.

To be more productive, we should focus on the 99%, not on the 1%. How can we learn faster?

Pareto and headache

Raganwald asks:

  1. 20% of the features are responsible for 80% of the headaches of software development, and;
  2. 20% of the features are responsible for 80% of the value of the software to its users.

Are those the same 20% of the features on your project? If not, why not?

I cannot decide if that is a rhetorical question or not, so here’s my answer:

At the beginning of a project, the two are unrelated. What happens to them during the project largely determines its success.

A good team quickly notices which features cause more headaches. They will ask if the expected value of these features is great enough to justify the pain. If not, they throw those features away. The longer the project lasts, the more aligned the values of the features are to the pain they cause.

There are at least two ways to fail in this. First, the team may not notice early enough that a feature is painful. When they do notice, the pain of an alternate path may have grown beyond the expected pain of finishing and maintaining the feature.

Secondly, the team may be unable or unwilling to choose an alternate path. They have a contract or a boss or a mindset that tells them to finish the software the way it was originally planned.

Unfortunately, in most projects I have participated in, there has been little overlap between the painful and the useful features. I believe the same holds true for the industry in general.

A word of warning. If you think this is an argument for agile development, think again. If you think this is an argument for waterfall development, think again, too.

Agile Web Development course report

I wrote a report about the course I taught at Tampere University of Technology in April: Experiences from Agile Web Development course, spring 2008.

Perceived learning per topic.

Top-down, Bottom-up, and Outside-in

(This is part of a post I wrote to Agile Finland mailing list. You can read the entire thread there.)

The three ways to design software

There are three approaches to software design: top-down, bottom-up and outside-in. (People mean different things when they say top-down, so don’t be surprised if my definition doesn’t match yours. What I am describing here should be taken as a clarification of the rest of the post, not as a normative definition. The same goes to bottom-up and outside-in.)

Bottom-up means you start with the “smallest things”. For example, you know that you are going to need a custom communication protocol for your distributed application, so you start by writing the code for that. Then you write – let’s say – database code and then UI code and finally something to glue them all together. The overall design becomes apparent only when you have all the modules ready.

Top-down starts with the overall design. You find modules and interfaces between them, then go on to design class hierarchies and interfaces inside individual classes. You go into smaller and smaller detail until you reach the code level. At that point your design is ready and you start the actual implementation. This is the classical sequential approach to software process.

Outside-in is an iterative method. You start with one user story, and write code for that. Then you go on to the next user story, and so on. You start implementing from the code that is closest to the user and proceed toward more and more implementation detail. In a web application (using TDD) this order could be: acceptance test, route test, route code, view test, view code, controller test, controller code, model test, model code. You only start writing code for a lower level when the upper level requires that.

The understanding – or fashion – today is that TDD is best done outside-in. It may be a little more difficult technically, since you need to isolate layers somehow, so that you don’t have too many failing tests in your green-red cycle. Typically mock objects are used for this. Of course isolating unit tests is a good idea regardless of your approach, so you may want to use mock objects anyway.

Architectural design

Now the important question of course is, how do you make sure that the overall design works with outside-in process. In my opinion it doesn’t hurt to do a little top-down design before you start the outside-in cycles. However, be careful not to over-do it. Using a popular framework helps. The top-down design phase can be very short when you have something well-known and well-tested as the foundation of your architecture.

After the rough design is ready, the first few user story iterations should strive to validate that architecture (especially if you have rolled out your own). Architectural redesign is cheap early in the project but very costly later on. Using outside-in is an improvement over a sequential approach in this sense. With a waterfall process you don’t get feedback until much later, and if the architecture doesn’t work, it will be usually too late to fix it.

Responding to requirement changes

Another important consideration is how you deal with requirement changes, as there will be some in every project. It is interesting to note the differences between the three approaches in this regard.

With bottom-up, we have small isolated libraries. The hope is that we can reuse them easily when the need arises. Also, a well used bottom-up design builds up the abstraction level in every layer. The top levels will be easy to understand and easy to change. This is the Paul Graham argument for bottom-up design.

Top-down takes care of extendability by carefully desigining extension points in the architectural design. Elaborate class hierarchies and design patterns are used to make reuse and modifiability as easy as possible.

Outside-in doesn’t concentrate on modifiability per se. The belief is that the best way to provide malleability is to write the simplest – or minimal – code possible. When there is nothing that is not needed, the purpose and the functionality of the code is easy to understand and easy to modify.

Why outside-in

Experience has shown that the top-down approach to modifiability doesn’t work. Designing for extendability makes the code unnecessarily complicated, slowing the implementation process. What’s worse is that the extension points usually end up in the wrong places. The requirement changes often come as surprise, in a place you wouldn’t expect. The original design didn’t allow for extendability after all, and an expensive redesign is needed.

Compared with bottom-up, the advantage of outside-in again is the minimality of the code. Writing a module with bottom-up approach it is hard to guess what is actually needed by the upper layers. The interfaces will have unnecessary classes and methods. This takes time, and makes the code harder to use.

Of course my discussion here is both simplified and polarized for the sake of the argument. In real life you will not make a clear-cut choice between the three approaches and stick with it, but you will use whatever approach you believe works best in each situation.