Best Agile Methods
Agile isn’t easy, and it isn’t the answer to everything, but when conditions are right it’s more effective than blindly trying to force software projects to behave like other, less flimsy sorts of projects.
One of the most important things to remember, if you decide to adopt an Agile methodology, is that one of its core principles is not dogmatically adhering to a rigid process. In this Tutorial, we uncovering 25+ ways of developing software by various Agile methods in 2019.
It’s important to find a process that works for your team, your business, and the problems that you’re trying to solve. If you find that one aspect of a particular Agile method works for you but another doesn’t always feel free to just take the bits you want and ignore the rest.
Looking, therefore, at big-A Agile, how do we get to some helpful specifics? Well, on the Agile Manifesto website there is a handy section called the Twelve Principles of Agile Software Development which gives a little more concrete detail. To summarize the already terse, the meat of the Principles is:
Deliver software “early and continuously”.
Embrace and encourage changing requirements and use them to your advantage.
Ensure the developers and non-developers interact daily.
Regularly evaluate and adjust the process to make it better.
Thankfully, however, there are several more specific disciplines that offer more practical advice on how to adhere to the Manifesto and accompanying Principles. To get a better sense of what Agile is all about, let’s take a look at one of the more popular disciplines that you might come across: SCRUM.
There is, however, another aspect of rugby gameplay, called a scrum. In certain situations, the referee will call for this, and eight players from each team will crouch into a tight formation three rows deep, shoulder to shoulder and face to the buttock.
These two blocks of flesh will ram into each other and try to push each other backward, with the ball placed in the center of the melee between them.
Whichever team controls the ball can either attempt to extract it and get back to running with it, or they can try to drive forwards, essentially remaining information with the ball secured somewhere in their midst while pushing their opponents all the way back to the end of the field.
The rules of rugby have been adjusted in recent years to make these long drives less common, but it is from this idea of a scrum that the SCRUM movement got its name. It’s not clear why they decided to capitalize on it, because it was never meant to be an acronym.
The rugby analogy doesn’t really stretch any further than the principle that “we all push forwards together,” so at this point we’ll abandon it and look at the details of how SCRUM works directly.
The key structure in SCRUM is a “sprint,” which has a fixed length, normally 2 weeks, but sometimes 1, 4, or 8 weeks, over the course of which a fixed series of events occur.
Software development is divided into a repeating series of sprints, sometimes ending at a specified project end date but sometimes continuing indefinitely.
Broadly, the Product Owner needs to make sure that at the top of the backlog there is at least one sprint’s worth of work that is very well defined in small manageable chunks but, because so far they don’t know how long each task will take, they have to err on the side of caution and spec up more tasks that are actually needed.
This whole process is often called backlog “grooming,” and is either done before the start of each sprint in a dedicated session or happens continuously in the background.
At the start of the sprint, the Product Owner and the development team (which should have between 5 and 9 members) meet to do sprint planning.
This meeting is run by the Scrum Master (a role that’s sort of the bits of project management that the Product Owner doesn’t do, like chairing all meetings and ad-hoc problem solving to keep the developers happy and productive).
The input of the Product Owner is confined to explaining and clarifying the details of the tasks in the product backlog.
The development team then collectively estimates the tasks in the product backlog, starting at the top and continuing until they have estimated enough work, according to those estimates, to keep them busy for the forthcoming sprint.
The tasks they estimate then form the “sprint backlog,” which is the forthcoming sprint’s to-do list.
Once the sprint backlog is defined, the sprint can start, and the developers take tasks from the top of the sprint backlog, work on them, complete them, and then take more tasks from the top of the sprint backlog.
Every day, normally first thing in the morning, there is a meeting of the developers, called the stand-up (I’ll let you work out why it’s called that for yourself).
In this meeting, everyone reports on what they were working on, what they will be working on, and what is standing in their way. It’s spectacularly easy to get sidetracked in these meetings, so the Scrum Master is there to remind everyone that the meeting is a series of reports, not a discussion.
Discussion points are followed up separately, particularly if they don’t affect the majority of the team.
Progress throughout the sprint is tracked, normally by some sort of cards-on-boards display, either physical or digital. Typically there is a “To do” column, a “Doing” column, and a “Done” column, and cards for each task get moved across as appropriate.
It is also common to have a sprint burndown graph, which tracks the total story points of all tasks that are not in the “Done” column at the start of each day.
The ideal graph is a smooth progression from the initial number of story points in the sprint backlog down to zero over the course of the sprint, but normally the graph stays pretty flat, until the last couple of days, when it topples down to somewhere a little above zero right at the end.
At the end of the sprint, a meeting is called for the development team plus anyone else who cares about what they are building—clients, bosses, end users, etc. This is the “sprint review,” which is a chance for outsiders to see what the team has been working on.
Typically the Product Owner talks through what is new since the previous sprint review, and then everyone has a chance to try the software out for themselves, following which the Product Owner gathers feedback.
The Product Owner uses this feedback and their own observations of the software in use to add to and re-prioritize the product backlog in the next backlog grooming session.
Finally, the non-developers clear out and the development team is left for the final meeting of the sprint, the “sprint retrospective.” The purpose of this is to assess not what was built but rather how it was built.
It’s an opportunity to suggest adjustments to the process, which the team may or may not agree collectively to incorporate into future sprints.
Another formal Agile practice you might come across is called Extreme Programming, or XP. I won’t dwell on it too much here because, as a discipline, it has more to say about the technical nuts and bolts of software development than SCRUM does, and arguably less about the broader process.
It takes to an extreme (hence the name) the idea of rejecting design at the start in favor of incorporating change as it comes along—while in SCRUM once a sprint is planned there can be no changes made to it, and new requirements can only go into the next sprint.
XP is much more flexible about incorporating changing requirements as they emerge because it considers current requirements always to be more meaningful than anything planned in advance.
XP also places a heavy emphasis on automated tests of the code, to the extent that all requirements must be expressed as a set of automated tests, designed such that they will only pass if the required functionality is built.
This means that the “customer” (i.e., whoever’s asking for the work to be done) needs to express their requirements by actually writing automated tests.
This is hard because automated tests are written using a programming language (although sometimes they use languages that look quite a lot like English, which helps), and the customer isn’t expected to be a programmer.
To get around this, XP dictates that the customer should be embedded into the development team so that they can work with the coders to put together the tests together, and be on hand as soon as any new functionality passes the tests to ensure that it does indeed meet the requirements that the tests were based on.
Roles in XP often have different names, and there’s a general rule that the team should have no more than 12 people in it.
If XP is much more prescriptive about the details than SCRUM, another popular discipline, Kanban, is the complete opposite. Kanban was actually one of a raft of methods developed by Toyota to improve their car manufacturing processes, but it has also proved pretty popular in the software world.
Eschewing sprints, defined meeting structures, and roles, the core idea of Kanban is basically as simple as a big board with some columns on it.
The column on the left is for tasks that have not yet been worked on, the one on the right is for tasks that have been completed, and the columns in the center are for each stage that an in-progress task might go through.
One person is in charge of prioritizing the items in the left-hand column so that the most urgent ones are at the top, and from then on the rules are simple:
When a developer finishes what they’re working on, they pick a task from the top of the left-hand column and move it one column to the right.
They are then responsible for working on it until it moves to the next stage, at which point the story moves to the right again, and either they keep working on it until the task makes it to the right-most column, or they hand it off to someone else.
Furthermore, there’s a strict limit to how many tasks can be in any one column at a time, so if a column is “full” then the developer must find a way of moving a task from that column on before they can move anything more into it.
The key metric is how fast it takes the average task to get from the left-hand column to the right-hand one. This “cycle time” is what’s used to predict how quickly large projects will be completed or milestones will be reached.
[Note: You can free download the complete Office 365 and Office 2019 com setup Guide for here]
The advantages of Agile
There are many benefits to the Agile way of working, whichever methodology is chosen. But for our purposes, the key thing to note is how it essentially sidesteps the three problems we identified in the previous blog.
The Imagination Problem crops up when we the customer try to define in advance the details of what we want to build and make plans based on that.
If we use SCRUM, all we need to do is define the details of what we need in the next sprint, and if we miss something that’s no problem—we’ll find out in the sprint demo at the end of the sprint, and can then add changes and adjustments to the requirements in time to be incorporated into the next sprint.
Using XP we could make those adjustments even as the feature is written in the first place because we the customer are embedded in the development team so we get to see the features as they’re being built.
The Agile way is premised on the assumption that planning everything in advance is impossible, and is therefore designed so that we’re not expected to try.
Similarly, the Estimation Problem crops up when we try to work out in advance how long everything will take. We saw in the previous blog that the best mitigation of the inaccuracy of software developers is empirical data, and Agile development provides this from a very early stage, in the form of velocity or cycle time or something similar.
So long as software developers are broadly consistent in the long run about how inaccurate their estimates are, which they tend to be, Agile provides mechanisms for compensating for the inaccuracies.
The caveat here is that Agile development still doesn’t offer any help when we need to put together timelines before development has even started, but we’ll come back to that one.
Finally, the Arithmetic Problem is only a problem if it is discovered, relatively late in the project, that the project is running late, and if the best available solution is to add more developers. (Let’s not forget that, in traditionally run software projects, this oddly specific set of circumstances comes to pass all the bloody time.)
Agile development provides both a weak and a strong defense against this problem. The weak defense is a baked-in disposition against adding team members.
Both SCRUM and XP set limits on how big a team can be, and place a premium on fostering a team identity specific to the people involved. Agile teams will automatically push back on chucking bodies at a problem.
The strong defense is that Agile methods make it harder for project slips to remain concealed until late in the project. If the team doesn’t move as fast as you thought, your cycle time will become apparent within the first few weeks.
If you didn’t realize that you’d need a bunch of extra functionality for the new software to be usable when you signed off on the project, it should become clear in the first or second sprint review when your end users try to play with the early prototypes.
If (or rather when) it turns out there’s a lot of technical complexity in wiring up the different components of the new system, it’ll be uncovered early on because the need to provide the customer with working software means that wiring up the components is one of the first tasks to be done.
Knowing early on that there’s a problem makes it significantly easier to adjust the plan, and if the correct response is to throw more bodies at the problem, it’s much, much better to do so at the beginning, so that there’s less built-up knowledge for the new team members to pick up.
These theoretical advantages to Agile development translate into measurable benefits. The 2015 Standish report has the success rate of Agile software projects at 39% compared to 11% for traditional, or “waterfall” projects. And for projects that aren’t entirely successful, only 9% of Agile projects are classed as outright failures compared to 29% of waterfall projects.
These results should be taken with a pinch of salt because there’s huge potential for reporting bias. Remember how in the last blog we discussed how large proportions of managers go into IT projects assuming they’ll fail?
Those managers are fairly likely to report a project as having been a failure when asked in an anonymous survey.
However, Agile managers tend to be fairly enthused about their approach—because it’s still relatively new, and promises to address the problems they’re used to experiencing over and over—and so they’re much less likely to start a new project with a fatalistic outlook, which will affect how they perceive and report the success of the project upon completion.
Nevertheless, the trend in the data is undeniable. Agile help. It is not, however, a panacea. A 40% chance of total success and a 10% chance of outright failure are still not great odds.
The rest of this blog will be devoted to exploring the issues that Agile projects run into and attempting to offer advice on how to navigate them.
Small sprints and big decisions
I once worked as part of a team that managed to thoroughly confuse our boss. We had a large consumer-facing product to build whose details were still being worked out, but it was vital for the survival of the company that we launched the product soon. Therefore, the boss was keen for us to start work on it straight away.
However, some of us refused, howling at him that we couldn’t possibly start building something until he’d worked out what he wanted us to build—if we guessed at it, we’d certainly get it wrong and we’d have done so much wasted work.
Worse, until we knew exactly what he wanted, how could we make the right big decisions about languages, frameworks, and tools?
So he then proposed that he’d sit down with the designer and put together a comprehensive spec document, complete with designs, and give it to us to build, only to find us howling at him again.
This time with some of us complaining that we were a modern, Agile team who didn’t believe in “Big Design Up Front,” as it’s sometimes called and that we didn’t have faith in a spec that was entirely written without any experience of a working product.
The spec would almost certainly get it wrong, and building from it would be so much wasted work! Eventually, he called us all into a meeting room, sat us down, and asked, in an exasperated tone, “So what is it that you actually want?”
There were two underlying issues. The first was that within the team there were developers (my younger self-included) who were foaming-at-the-mouth devotees of SCRUM who hated the idea of a big spec on principle.
But there were also developers who weren’t true believers in the Agile Way, who hated the idea of not having a big spec on principle.
The solution to that was simply to get us all to agree to a single way of working, and because our tech lead was SCRUMmy, that’s the path we chose.
The second underlying issue, however, is the more interesting one. The complaint that the initially non-SCRUMmy developers made (that it’s hugely inefficient to start building something before you know what it is you want to build) was an entirely valid one.
And just because once we all started using SCRUM we stopped flagging it as a problem doesn’t mean that SCRUM, or Agile in general, negate that sort of problem. This is because, in absolute terms, Agile development isn’t a very efficient way of working.
As we discussed in the previous blog, the change in software is expensive. A changed requirement necessitates rewriting the code that was written to fulfill the original requirement.
But it also can necessitate swapping out a library or framework that one chose because it suited the original requirement, and doing that can be like stripping out the electrical wiring out of a building and replacing it: it’s time-consuming and disruptive, and doing it can cause unexpected things to happen or, worse, to stop happening.
But Agile methodologies actively embrace change. When working out what to build first when working in an Agile way, one doesn’t say, “What is the component I’m most confident will not change?”
Rather, one is encouraged to say, “What is the thing that could alter my preconceptions the most if I can get to try out a working prototype of it?”
In other words, one deliberately dives into the most uncertain areas, takes a guess as to what might be the right thing to build, and builds that just to see what it’s like, using that experience to try to inform some decisions about what the actual right thing to build is.
If that’s one’s approach, it’s almost guaranteed that the bit that gets built first absolutely will change, and quite possibly more than once. Not only that, but there’s a fair chance that the change will be radical.
If you start building a house before there’s a complete design, you might try building it out of bricks only to discover a few weeks into the process that what the client really wants is glass and steel, in which case you’ll have to rip down your initial walls and start again.
It’s not unheard of for Agile development to have to start rebuilding from scratch as a result of feedback from early prototypes.
If this sounds terribly inefficient, that’s because it is. Change is inefficient, and Agile encourages change. But Agile practitioners, knowing this, still keep doing what they do. It’s like that almost-certainly-apocryphal quote attributed to Winston Churchill: “Democracy is the worst form of government, except for all the others.”
Given that eliminating change from software projects has proved to be impossible, due in large part to the previously discussed Imagination Problem, it turns out that methods that encourage change and deal with it well are more efficient than methods that try (but inevitably fail) to eliminate it and therefore fail to accommodate it.
Keeping it minimal
There are, though, ways of reducing the disruptive violence of change— not resulting in less change, just in the change being less painful.
A team of developers who expect change can select tools and technologies that are versatile rather than those that are ideally and narrowly suited to the probably-wrong initial definition of what the job is.
A key word here is “modularity.” If you buy a scythe and you then find out you need a rake, the ability to retain the handle and simply swap out what attaches to its head can save time and effort.
So too in software, when the code has been written in a modular way and using modular libraries, swapping one thing in for something else becomes a less miserable task.
There is, to be clear, an up-front time cost to making code modular (in the same way you’d expect a scythe with an easily detachable head to cost more than one with a fixed head), but the overall time-saving in an Agile project can be significant.
The other important thing when anticipating change is to avoid building anything unnecessary. As a developer, if I was asked to build a website that loads some text from a database and show it on a web-page, I’d be sorely tempted also to build what’s called a “caching” mechanism.
So that if the website receives lots and lots of visitors that doesn’t cause it to slow down too much. This is an example of me “optimizing” my code, making it more resilient to extreme circumstances.
If I know that the website will receive lots and lots of visitors than by building the mechanism straight away I’m saving time, because if I come back and add it in later then I’ll have to remind myself of how the code works so that I can plug in my caching mechanism in the right way, whereas while I’m writing it for the first time it’s fresh in my head.
If the website never receives lots and lots of visitors then I’ve possibly wasted my time, although I could argue that it’s better to be safe than sorry—insurance policies aren’t a waste of money even if you never happen to need to make a claim.
Whereas if, after I show my client a prototype of the website they asked me to build, they decide that actually they don’t need to store the text in a database at all but rather I should simply embed their Twitter feed on the page.
Then my work to set up a caching mechanism has 100% incontrovertibly been a waste of time. (Note that the rest of the work to build the prototype wasn’t a waste at all—it was what I needed to do to allow the client to realize that what they needed was a Twitter feed.)
If the likelihood that the client will change their mind about what they want is high enough, the likelihood of a caching mechanism being a waste of time counteracts the benefits that a caching mechanism would bring if the client didn’t change their mind.
The temptation to optimize felt by developers is strong because we want to build software that’s as good as possible, and normally optimization makes software better. However, it can lead us to waste effort when we lose sight of what we’re working towards.
When it comes to Agile development, what we’re working towards should always be the next memorable prototype. The key realization (one that managers would be well advised to help developers to make) is that we are not trying to build the final product from the beginning.
Rather, we are trying to put together an experiment to help us determine what the final product should be. We developers should judge our software based on how well it supports that experiment.
If the initial experiment is “Does having a web page that loads text from a database and displays it make the client happy?”, then optimizing the code to enable lots and lots of visitors to view the same page at once is clearly irrelevant.
Later down the line, if we become confident that the final product will indeed involve loading text from a database, we may agree to talk to the client about enabling large numbers of simultaneous visitors.
And the experiment may become “Does adding a caching mechanism make the client happy with how the website behaves under a heavy load without introducing unacceptable tradeoffs?”
At that point, our work on caching clearly does support that experiment.
The temptation to do more work than is strictly needed and the possible negative consequences are well known in the software world—it’s a hackneyed adage that “premature optimization is the root of all evil.”
It has led the Agile community to place a high value on a particular form of laziness, specifically the habit of only doing the bare minimum amount of work to fulfill requirements. It turns out this sort of laziness is a very good thing because in a world of change it minimizes wasted effort.
But it’s quite hard for developers to be this lazy—we’re a proactive bunch normally—so XP, in particular, goes to some lengths to enforce it. XP dictates that developers must use Test Driven Development or TDD. TDD mandates that first, one writes a test that describes one facet of how one wants the software to work.
Since the software doesn’t work like that yet, the test fails at first. The developer then does the smallest possible amount of work to make the test pass.
Then they move on to the next test, which describes another facet. In XP the tests are provided by the “customer,” meaning that they describe only those things that the customer wants to see in the current sprint. If the customer doesn’t explicitly ask for a caching mechanism, there won’t be a test for it.
The TDD process should prevent the developer from getting overly eager and building it anyway. And even if they do, there should be some oversight from a second developer, from either code review or pair programming, making sure that no naughty premature optimization slips through the cracks.
All of this is to say that the initial problem with Agile development that we described in this section—“How can we make the big decisions about what technologies and architectures to use if we don’t have a complete set of requirements?”—is indeed a problem.
The answer is that we can’t. But the Agile way is to try not to make big decisions wherever possible.
Rather, focus on the small decisions of: “What technologies and architectures should I use to make the next prototype, bearing in mind that I’d like to be able to reuse as much as possible when making the prototype after that, which might be entirely different?”
As with most waterfall projects, the upshot is that the technology and architecture choices made at the start of the project will probably change.
But in an Agile project, the impact of that change will hopefully be lessened, so long as less unnecessary work is done that will have to be thrown away, and all technical decisions are made anticipating change.
As a non-technical manager, you aren’t directly responsible for either of these, but it is your responsibility to remind developers periodically that change will happen, and ensure that they’re approaching their work accordingly.
The next big problem with Agile development is that it can be quite hard to fit an Agile team-shaped peg into a traditional business-shaped hole. Suppose you decide to follow SCRUM with your team.
As the team’s manager, that might feel like your decision to make, since it primarily affects the working practices of your direct reports.
The problem is that SCRUM is reliant on external parties in several ways—by adopting SCRUM you’re placing a burden on people outside your team, and if they don’t fulfill their new responsibilities, the whole thing might fall apart.
In this section, we’ll look at some of the ways in which stakeholders resist buying into the process, and the things that can go wrong if you don’t get buy-in.
“I don’t need to check in every week—just send me a report”
Important people are busy people, and in a large business, it’s normally important people who cause software projects to happen. Occasionally those important people don’t have much of an interest in the software that gets built—they were merely the ones to greenlight something that someone less important asked for.
But more often than not the important people do care about the outcome of the project, and want to be able to have oversight of what gets built. In a traditionally managed project their input might be solicited at the beginning, to sign off on the spec, and then at the end, to sign off on the software that should meet the spec.
But in an Agile project, as a major stakeholder, they’re requested to attend every sprint review (or equivalent). This can easily mean an hour of their time, every week, for the duration of the project.
Unsurprisingly, busy people—quite reasonably—don’t like meetings that don’t feel strictly necessary. And sprint review meetings don’t always feel strictly necessary.
A common response is, “Well, I’m not actually a part of the team, so I shouldn’t need to come to all the team’s meetings. Can’t you just send me a regular email to keep me updated on your progress?”
The answer is no, absolutely not. The sprint review is not for the benefit of the stakeholder; it’s for the benefit of the team. It’s not an opportunity for the stakeholder to find out what the team is up to, but rather an opportunity for the team to find out what the stakeholder actually wants.
Since there wasn’t a complete spec signed off at the start of the project, the team only gets to find out what to build next through the regular input from the stakeholders.
The deal you have to get stakeholders to agree to is that they can request changes and additions at any point over the course of the project, but only at the regular reviews. If they don’t turn up, they don’t get a say.
“But I already know what I want”
This is a common follow-up to the first objection. Often the person who requests that the work is done in the first place already feels like they have a pretty clear idea of what they want, and feel it would be far more efficient if they could simply write it down, hand it over, and let you get on with it.
If this is how your stakeholder feels, and you want to work in an Agile manner, you may need to find a way to refuse their request, but your refusal is going to have to be couched in the most diplomatic terms.
That’s because it’s probably premised on the assumption that anything the stakeholder says in advance about what they want, based on their mental image of what the software and accompanying process will be like, is fallible, and that the only way to get a complete truth is to periodically present the stakeholder with working software.
So that they can revise their assumptions based on that software rather than their faulty imagination.
But clearly, you can’t say to your boss, “I don’t trust you to know what you want before I show it to you”. That way promotions and pay rises do not lie. And your colleagues in other departments won’t be best impressed either.
It can be more helpful to phrase the refusal something like: “In this project, I anticipate lots of unforeseeable edge cases, and we’re going to have to navigate how to handle them as and when they occur.
As there’s a chance one of them will cause us to have to re-think some aspect of the user flow, getting your continued input and guidance over the entire course of the project would be more valuable than just getting your input at the start.”
That being said, if someone wants to provide you with a comprehensive spec, sometimes it can be easier to just let them. You then pick which elements to build first from that, and then start your Agile process and quietly allow the original spec to be forgotten about, so long as the stakeholder provides more updated requirements at sprint reviews.
“But this new thing needs to get done right now”
This is a scenario where the advertised advantages of Agile can really bite you. In a waterfall project, change is a serious matter. New requirements mean changing the plan, and that normally involves paperwork, the goal of which is basically to discourage anyone from attempting change because, as we’ve discussed, it’s disruptive and painful.
If you’re being Agile you recognize that change is often necessary, and you welcome it, while trying to focus more on making change less painful.
Often, though, the mechanisms that exist to reduce the pain of change get conveniently forgotten in the heat of the moment.
When a new requirement comes in (e.g., “We have the opportunity for a potentially very lucrative partnership with X company, which means our priority is now demonstrating to them feature Y in action at the Z meeting”), there can be a presumption that the team will drop everything to accommodate the request.
“After all, what’s the point of an Agile team who can’t respond to change? That’s what being Agile is!”
The difficulty is that being Agile isn’t about being willing to drop everything at a moment’s notice and jump into a change of direction. Different Agile methods have very specific processes for dealing with change.
SCRUM, for example, makes it a golden rule that while each sprint can veer off in a completely different direction from the last, you never move the goalposts of an in-progress sprint. Depending on the length of the sprint, that can mean up to 4 weeks before a radical change of direction can be accommodated.
There are several reasons for this. One is that the predictive power of velocity calculations is reliant on stability—take that away and it’s much harder to gather relevant data about how quickly the team moves, which hurts in the long term.
Another is that change without due planning leads to clumsily structured software that becomes harder to maintain over time, which hurts in the long term. Yet another is that it’s very dispiriting for software developers never to be allowed to finish anything, and poor team morale hurts in the long term.
All of these are reasons why, in general, changing without due process is a bad thing even when the thing being changed to is a very good and/or important thing. The difficulty is that in each specific case the benefit of changing just this one may seem to outweigh the general problems it will cause. Beware of this sort of reasoning.
When my friends and I were young and foolish and trying to make our way in the big city (London in our case) in our early twenties, I had a friend who had a habit of taking taxis home after a big night out, eschewing the 24-hour “night bus”
On the grounds that while it cost a twentieth of the price, a taxi would take up to an hour off the journey time and would involve less dealing with drunk crazies and other people’s vomit. And of course, “it won’t break the bank to take one taxi.”
Now at 2 am on a cold October night in Trafalgar Square that’s a pretty compelling argument, but after a couple of years of late-night taxis, you can guess which one of our gang was struggling with credit card debt and bitterly regretting all those 2 am choices.
If you possibly can, be firm about sticking to the processes for managing the change that you’ve adopted. If it becomes clear that those processes don’t work for your situation, you can absolutely change them, but do so after discussion and reflection, rather than dropping everything as a knee-jerk reaction to a crisis.
“But I need those estimates now”
As discussed in the previous blog, estimates given at the start of a project are normally fantastically unreliable, and Agile development focuses on waiting until there is empirical data before making predictions about the future.
However, business people are used to getting estimates at the start of a project, and often they are reliant on them—if, for example, they need to coordinate software completion with activities in other parts of the business, believable estimates are indispensable.
Therefore if at the start of the process you refuse to provide estimates of how long getting to completion will take, you can upset people. The old “I can’t tell you how long it will take to build what you want because I don’t believe you know what you want” line doesn’t go down very well, as you might expect.
And often neither does the “I could give you estimates now but they’re almost certainly wrong because I don’t know how much to compensate for my team’s over-optimism” line—it makes it sound like you think your team is idiots and you don’t communicate well with them.
Often people would prefer the illusion of knowledge by being given unreliable estimates to the honest truth that it’s often impossible to give reliable estimates at the start of a software project, particularly if it’s a new team. So what can you do?
The first option is to fudge it and say, “I can give you an unreliable but broadly indicative estimate now and a more confident prediction later, once we’ve seen how quickly the team can move.” Then pick a number, multiply it by the total number of days your developers say each task will take, give ’em that and forget about it.
This can work, but only in situations where it’s genuinely understood that the number is just the best guess. All too often your caveats will be ignored, and you’ll find yourself being held accountable for missing deadlines based on the estimates that you yourself said were unreliable.
The second option is to be more pragmatic and say: “I can’t yet tell you when the project will be complete, but if you really need a date to organize things around, I can commit to saying that we will have something shippable by X.”
This approach is less risky than it perhaps sounds because the focus in most forms of Agile is on getting something at least potentially shippable at the end of every iteration. The need to have something that can be demoed at regular intervals means that you just do end up with something shippable faster— even if it doesn’t do very much.
In a traditional project, you might work by first building a full-featured user interface that doesn’t actually do anything, then building a full-featured backend that doesn’t have a user interface, then trying to connect the two together.
Until you’re finished connecting the two, you don’t have a working product you can ship. And if connecting the two is delayed, your earliest possible ship date is delayed.
Whereas the Agile way is to first build something very small that has both a user interface and a back end, that does something but not very much, so that you can give a working demo of it. Then you add a new bit to the user interface with matching back-end functionality, so you can demo that. And so on.
Which means that, while it might take the same amount of time to complete every feature in the initial specification, if things run slow and you run out of time, you can simply ship the software at the state it got to—it won’t be feature-complete, but it will be better than not having anything shippable at all.
Thanks to this, you are less likely to be really screwed if you say “I’ll have something working by X date” and then things take longer than expected. In this way, if people really want to squeeze dates out of you, you can oblige without being disingenuous.
Buy-in is fine, but embraces are better
In my experience, the best way to get buy-in from external stakeholders in your organization is to get the people excited about Agile as a process. It’s still comparatively new, it’s still comparatively trendy, it’s got hard data demonstrating its effectiveness.
And adopting the latest Agile techniques is something you show off to your peers with at corporate networking events. Therefore people will often respond positively to being invited to take part in your Agile process if you tell them that that’s what it is.
That being said, often they will have a hazy and/or utterly inaccurate picture of what that means (because, as previously discussed, everyone is agile, but few are Agile), so it will be up to you to educate them about what your process is.
Make the time to explicitly explain it, either in a face-to-face conversation or through a presentation at the start of the project (don’t email a memo; no one reads memos). If people embrace the concept they’re much more likely to play along when the going gets tough.
Embedded designers and the two-way conversation
One very common issue with Agile development is the question of what to do with designers and the design process when the software involves a user interface.
In this context, I’m talking about two sorts of designers in particular: the UX (User Experience) designer, who determines how the user can interact with the interface, and the graphic designer, who determines what the interface looks like.
If you’re lucky enough to have a separate person in each of the three roles of “customer” (remember that in Agile-speak, a “customer” may be your boss or another colleague—it’s just whoever is asking for the software to be built in the first place), UX designer, and graphic designer, the process may work something like this: The customer says, “Our users need to be able to email a link to their friends.”
The UX designer says, “OK, we need a ‘Share’ button in the top right, and clicking it should copy a link to the clipboard and a message should appear saying ‘A link has been copied to your clipboard; paste it into an email or message to share it with your friends.’”
The graphic designer then draws up an image of what the button and pop-up message should look like.
All this gets passed on to the development team, and together the initial requirement plus the UX and graphic design define exactly what the developers build. However, it’s not as simple as it sounds.
Often the designers work essentially separate from the development team. This presents two problems. The first is that it can be difficult to get the design in sync with what the development team needs.
If the designers churn designs out too slowly, the developers find themselves waiting around because they can’t start work until it’s clear what they need to build.
If the designers work too fast and provide a complete set of designs at the start, then the chances are they’ll base their designs on requirements that will later change, and before their designs are even built they will need to be redone at least once as the iterative process evolves the customer’s understanding of what actually needs building.
And even if new designs are coming through once a sprint, getting them to come through at the right time during the sprint cycle (i.e., after the requirements for the sprint have been locked down but before the developers start working on building the UI) is very, very hard.
Two steps forward, three steps back
The second problem is a more practical one. Suppose you manage to get your design team working separately from, but in parallel to, your development team, sending you designs for each feature just in time for when it’s needed.
Now, suppose that in sprint 1 the user interface is very simple—let’s suppose all you’re building is a box to enter text and a button to submit the text that’s been entered, plus a panel to show the history of the previous messages that have been entered.
Here’s a common occurrence: The developers look at the designs and realize that there’s a fantastic tool called “Message Lister” that they can use—it’s open source, permissively licensed and one of the big things it does is provide a pre-built user interface with a customizable text box, button, and message history pane. Great!
The developers can match the designs by customizing Message Lister, saving them loads of time, which is just as well, because that other task, to hook up the message history to a database to store the text entered for next time, took longer than anticipated.
The first sprint is successfully completed, and at the sprint review, everyone has a chance to see their design in action. The designers, Product Owner, and other stakeholders come to the conclusion that it’s really important that the user be able to see, for past messages, the date at which those messages were sent.
Between them they come up with a more nuanced design for the second sprint: now they want to group past messages by which day they were sent and add a label for each day, as well as several other new features that make sense now that they’ve seen the first prototype in action.
And this is where things get sticky. The developers realize that Message Lister simply doesn’t allow them to group past messages by day, even though it can label each message with its sent date.
They’d have to spend ages customizing Message Lister’s source code to be able to match the new designs, and frankly, at that point, it’d be just as quick to rebuild the user interface from scratch not using Message Lister at all.
The from-scratch UI option seems to offer more flexibility in the future, so that’s the route they go down, meaning that the bulk of sprint 2 is now dedicated to that.
The work goes well, and by the end of the sprint, they’ve managed to rebuild both the message box and the message history panel to match what they previously had at the end of the first sprint, only this time it doesn’t use Message Lister.
But they didn’t manage to get around to the submit button, and they haven’t even touched on actually grouping past messages by date—it’s just that now they have the capability to add that next time, without any further rebuilding.
In other words, the prototype at the end of sprint 2 is actually less feature-rich than the one at the end of sprint 1. There’s almost no point in even holding a sprint review because all stakeholders have already seen everything the developers have to offer and more. It’s pretty dispiriting because it feels like the whole project has taken a step backward.
What I have just sketched out is a surprisingly recurrent scenario, and it illustrates the cost of change, particularly the cost of the very sort of change that an iterative, Agile approach is supposed to encourage.
Is this an argument for Big Design Up Front? No. As we’ve seen in the previous blog, BDUF works in theory, but the theory isn’t worth very much when your project is 2 months behind schedule.
One could also argue that this sort of change pain is an argument for delaying all visual design work until the very end of the project.
If you could wait until the requirements were stable before worrying about what the interface looked like, you could be confident that you’d only have to build the UI once. After all, you don’t paint a wall while you’re still building it.
However, this approach doesn’t really wash. First of all, the UX is one of the main things you’re trying to learn about through an iterative process. Trying to build a bunch of functionality first and retrofit a decent user experience on the top is nigh on impossible—just look at any recent version on Windows and you’ll see what I mean.
Second, I’ve emphasized the importance of having a potentially shippable product as early as possible in the Agile process. If you put off the graphic design until the end, then you’re dooming yourself to having to get to the end of the project before you can ship anything, which will hurt you if the project ends late.
There is, however, something you can do to mitigate the pain of change in this scenario, and it’s called dialogue. In this case, it would be perfectly possible to fulfill the new functionality requirements without rebuilding the UI from scratch.
This is because the actual functionality requirement was that users be able to see the date that each past message was sent. It was an interpretation of this requirement by the designers that caused the messages to be grouped by day in the new designs.
Once they’d explored the limitations of Message Lister, imagine if the developers went back to the UX designer and said, “It’ll take a day to add date labels to all past messages. But it’ll take an entire sprint and an extensive rebuild to group messages by day. Is grouping them like that worth the effort?”
Now, at this point the UX designer might insist that yes, it’s absolutely fundamentally necessary—they are the experts when it comes to user experience, and there might be a very good reason why grouping by day was specified.
But they might say that no, actually 90% of the value comes from having dates at all and that the grouping was just a nice-to-have that’s not worth it. In which case, this short verbal exchange might have just saved an entire sprint.
The truth is that this sort of trade-off is a fundamental part of software development. You’d be amazed at the amount of flexibility that appears in even the most rigid-seeming requirements when the specific time costs become apparent, particularly when those time costs would mean canceling or deferring other, quicker wins.
On the large scale, it is the Product Owner’s job to take into account both the value of a feature and it's time cost when deciding the priority of the product backlog, which determines what goes into each sprint. But even within a sprint, the possibility for trade-offs is always cropping up. Often they’re related to the graphic or UX design.
But you’ll often find there’s flexibility in the initial requirements as well if you ask the right questions. One of the smartest CEOs I ever met made it a matter of policy always to look for any possibility of a time-saving trade-off and run it by the stakeholders before accepting any user requirement into a sprint backlog.
In my experience, the best way to identify and capitalize on these tradeoffs is to embed the designers into the development team (and even, as in XP, to embed the other decision-maker, the customer, into the team as well).
If you do this, then instead of having the design for a feature specified at the start of a sprint, you can make designing the UI part of the sprint.
If you do this then you no longer have to worry about getting your design team in sync with your development team, to make sure the designs appear in time.
And having the designer working alongside (ideally physically alongside, but at the very least in the sense of working on the same thing at the same time as) the developers means that when it becomes clear that a particular design will take much longer than the designer expected, the dialogue can take place there and then, and appropriate tradeoffs can be made.
Having the designers as actual members of an Agile team as opposed to external stakeholders is by no means a new idea—you’ll find it listed as a requirement or even as an assumption in lots of blogs and articles on Agile technique.
But it’s one of those ideas that frequently gets ignored, particularly in companies where there is a separate design department because it sounds like it’ll cause disruption to report lines and payroll-based departmental budgeting.
If you possibly can, embed your designers into your team—it means that your design requirements become two-way conversations, and the flexibility that allows can save sprints.
Agile vs Lean
A word that goes hand in hand with “agile” is “lean”—particularly in start-up circles, where everybody wants to be lean because it sounds cool and trendy and unlike what “corporate” corporations do.
Unfortunately, there are several definitions of the term in different contexts, ranging from “something wishy-washy we say when we’re trying to convince investors we know what we’re doing” to “the specific techniques that Toyota developed alongside Kanban to optimize their production processes”—because, yes, Toyota came up with lean too.
I’m interested in one specific use of the word “lean,” which is the definition as put forward by Eric Ries in his seminal book The Lean Startup, the bible of many modern entrepreneurs.
I’d like to take a moment to discuss how it differs from but coheres nicely with, Agile software development. In the next section, I’m going to use a capitalized “Lean” to refer to Riesy leanness.
The first thing to be clear on is that Lean development is a process for building products, while Agile development is a process for building software. Which is to say, you can use Lean techniques for building products that involve no software, and you can use Agile for building software that isn’t a product, but not vice versa.
The fundamental idea behind Lean product development is as follows: If you have an idea for a product you want to build a business out of, your objective will be to work out the details of the product, it's marketing, and the business model around it to make it successful and profitable.
However, at the start, when you just have your initial ideas and assumptions to go on, is the worst possible time to be making decisions about the product, how it is marketed, and how it is monetized.
This is because you have no data to prove or disprove your assumptions, and most people are wrong most of the time when it comes to planning such things in advance (this emphasis on the fallibility of preconceptions, you may notice, smells a bit similar to the Imagination Problem).
The correct approach so says Lean, is to focus, with obsession and zeal, on gathering data.
Therefore, rather than build a fully-featured product, one should focus on building the “minimum viable product” first, or MVP. The MVP is the absolute bare minimum that you need to build to be able to put something out into the real world and start accumulating data.
And in particular, you should have specific data in mind when building your MVP and make absolutely certain that it will allow you to measure it.
Let’s look at an example. Suppose that I, as a budding entrepreneur, come up with a business idea. I notice that most people who own vacuum cleaners don’t use them most of the time. So I wonder if I could create a system to allow vacuum owners to rent out their cleaners to their neighbors via an online marketplace. I could call it U-Suck.
If I were to try to launch this idea as a business in a non-Lean (tubby?) way, I might build an online marketplace for vacuum lenders and borrowers, stick in place a pricing model, and then start publicizing my site to start attracting users.
The problem, which Lean types would be quick to point out, is that I don’t really know the first thing about what the marketplace should be like.
I might have my own notions, but without hard evidence, my notions are going to be very unreliable. I might spend a lot of time and money building something that turns out not to be at all well-suited to the desires and behaviors of its target users. So the goal should be at all stages to acquire evidence.
The first thing to find out is whether this idea even appeals to potential customers— vacuum owners and vacuum-needers. For my business to scale I’ll need to raise investment, and investors will want to know that my potential customer base is big.
So I need to prove that a large number of people are at least interested in the idea—if so, then I can work out how to “convert” those people into customers; if not, then I can save myself a lot of time and effort by giving up straight away.
Given the hypothesis that Joe Public is interested in the idea, what I need now is an MVP. The MVP is the simplest thing required to test that hypothesis by gathering relevant data.
In this case, the simplest thing is probably as simple as a web page somewhere that says “Coming soon,” and an email campaign sent to a mailing list comprising friends, family, and former colleagues, etc., with an explanation of the idea followed by a link to the web page.
If you track (as most email campaign software lets you) the ratio of link clicks to email opens, you can get a percentage figure of how many people are interested enough to at least click a link that either supports or disproves your hypothesis.
Rather than plan and build a product, you could put together this experiment in a matter of hours, practically for free.
The next step might be to find out whether the people who are interested are potential vacuum-borrowers, vacuum-owners, or (hopefully) an even mix of both, as knowing this will tell you where you need to put the most effort into improving the product proposition.
Perhaps you could add a button on your website that says, “I have a vacuum” and one that says, “I need a vacuum,” and see how many clicks each gets.
Once you know that, you can start figuring out how much people would be prepared to pay to rent a vacuum. You could add to your website another button that says “Rent now for only $5/hour” and see how many people click it, then the next week change the text to $10/ hour and compare the difference.
And so on. Note that by this point you will have learned a lot about your business before you’ve really committed any energy to build an actual online marketplace.
But note also that you have in fact already started building and delivering features. A website with 3 buttons might not feel functionality-rich, but that’s fine—you’re interested in a data-rich product, not a functionality-rich one.
If you carry on building, driven by hypotheses and MVPs, you will still end up with a complete product. Your trajectory might look as follows:
Question: “Will potential lenders be interested enough to sign up?” Experiment: Build a sign-up form that emails you user details and their vacuum availability.
Question: “Will potential borrowers actually request a vacuum?” Experiment: Manually update your website to list the location of each lendable vacuum and instruct borrowers to call you with their requests.
Question: “Will conversion increase if borrowing is a simpler, more automated process?” Experiment: Update the website to allow borrowers to select a local vacuum, enter credit card details, and a request date and time for collection and return.
Agile AND Lean
The Lean way is to work in a series of really tiny steps, at each stage saying, “If I do this, will it increase some number that I care about?” (That number could be conversion rate, customer base, customer satisfaction rating, etc.— whatever it is that your business needs or values.)
And I bring it up because while this is not actually the same as Agile software development, it ties into Agile really well. Both Lean and Agile rely on working in short iterations and incorporating the feedback of one iteration into the next.
Now the “feedback loop” for Lean is longer than in, say, SCRUM, because in Scrum you get all your feedback from a 1-2 hour sprint review, whereas in Lean product development you need to gather customer data, which might mean putting a new feature onto a website and watching it for a week or two to see what effect it has.
But that can be OK; it just means that the data from the features built in sprint 1 gets gathered while sprint 2 is in progress, and gets fed back into new requirements in sprint 3, and so on.
There is also a shared awareness of the importance of being minimalistic— when being Lean one only builds the minimum features needed to gather actionable data.
When being Agile one only writes the minimum amount of code to complete the feature as defined. Combining the two ways of working makes minimalism a universal and omnipresent value, which in turn makes it harder to get carried away and forget to be minimalistic, which is potentially a big pitfall for both approaches.
Finally, Agile’s receptiveness to change complements Lean’s experimentalism. The thing about experiments is that they’re as likely to disprove your hypotheses as they are to confirm them. This means that setting up every new experiment may mean undoing what was set up for the previous experiment.
Agile, as we have discussed above, provides frameworks for making the cost of that sort of change less painful. In short, I put it to you that if you are building a software product in a Lean way, it makes a tremendous amount of sense to be Agile about it.
When not to use Agile
I’ve talked about the benefits of Agile software development and some of its challenges. However, I would not claim that it is the right tool to use in every situation.
Agile is hard work, and it relies on an environment that satisfies a broad set of requirements to be effective. To finish this blog I’m going to briefly examine some situations where it may not be the right tool for the job.
Long cycle times
Agile development is based on cycles of development and feedback. The regular feedback from each cycle is used to define what gets built in the next cycle. If you don’t get that feedback regularly and often, you won’t know what to build.
If you have to wait for ages for feedback, you can only operate in very long cycles, and this can mean not getting enough input on what to build to be confident you’re building the right thing.
In a situation like that, you’d actually be better off working from requirements specified at the start, because probably-inaccurate requirements defined at the start may well be better than definitely-incomplete requirements derived from occasional feedback.
There are a few potential causes of long cycle times. One can be a dependence on hardware. When prototyping physical devices that include electronic components on which software can be run, the time to get from one prototype to the next can often be measured in months. If the software and hardware are deeply intertwined in a project, it can be a nightmare to manage the software in an Agile manner.
You might find, for example, that you can’t do a decent sprint review to gather feedback because you don’t have a working hardware prototype to run your new software on, and you won’t have one for another 2 months.
There are ways around this (you might be able to build software tools that allow you to simulate the hardware, etc.), but sometimes the time costs of the workarounds negate the benefits of Agile development.
Another common cause of long cycle times is busy stakeholders. If you operate in 2-week sprints, but your key stakeholder is only able to come to one sprint review in every four due to diary clashes, your effective cycle time is 8 weeks, not 2. 8 weeks is an awfully long time in which to veer off from what the stakeholder originally wanted.
In a scenario like this, building a complete spec at the start is perhaps more realistic than trying to get regular feedback.
The communicable and the knowable
Agile development mitigates against the unanticipated complexity that scuppers estimates, and the poorly imagined human processes that scupper specifications.
If you’re in a situation where you can avoid either of those in the first place, then you may find that Agile provides more hassle than a benefit—it is, after all, quite heavy on time-consuming meetings, and its active encouragement of change is less efficient than a system where change is genuinely avoidable.
Such situations do exist. Typically they involve software projects where a large part of the project involves repetition of something that has been done before, such as setting up a known system for a new customer.
If the vast majority of new code that needs to be written is merely about configuring something rather than adding any new functionality, and much of the project isn’t about writing code at all, then you may find yourself in a situation where you’re not really vulnerable to the Estimation Problem in the first place.
And then there are the occasional projects where the requirements are entirely clear, with absolutely no chance of changing. These can be hard to spot because people who want some software written often genuinely think that their requirements have no possibility of changing, and this has a very little bearing on the actual changeability of the project.
You have your best chance of avoiding change if the software to be written is not intended to enable a new human process.
Better still is if it’s something that humans won’t directly interact with at all—if it’s purely a component that sits between two or more automated systems. That way the Imagination Problem has less freedom to mess with the initial requirements.
Broadly, if you can possibly avoid Agile development without incurring significant risks, consider using something more direct and efficient. However, be aware that there are comparatively few software projects that are low enough risk to make it safe to avoid Agile.
Two types of trust
A big part of Agile is the empowerment of the team to make its own decisions about how to work. In particular, the person who would normally take on the role of the project manager in a traditional project may find themselves elevated to Product Owner status, and this entails having much more control over what gets built than a project manager.
The Product Owner interprets the input from the stakeholders and prioritizes them, which gives them the effective power to override the requirements of any stakeholder (although any PO who wants to remain in their post will exercise this power with extreme tact and caution).
If you find yourself in the Product Owner position and you have a boss who is remotely opinionated about the product your team is building, things won’t work at all unless your boss trusts you. But more than that, they have to trust you in the right way.
There are, I firmly believe, two types of trust that can exist between someone who wants something built and someone to whom they delegate responsibility for the building.
Both can be expressed naively as “I trust you to build me something good.” The devil, however, is in the detail, and it’s the subtle difference between both types that gets people in trouble.
The first type of trust expressed more fully, is, “I trust you to build me something good according to my definition of good.” The second is, “I trust you to build me something good according to your definition of good, even if that definition surprises me.”
The first type of trust—narrow trust—is trust that you will be given what you want, and the second type—broad trust—is trust that you will be given what you need.
This little distinction is often ignored, and it plays havoc in a collaborative project, particularly in the creative industries. Imagine if a film director says to a film composer, “I want some scary music for this scene, and I’m thinking of something with lots of percussions. Write me something—I trust you to make this scene sound awesome.”
The film composer either has a quite easy task or a very hard one, depending on what sort of trust the director is talking about.
If the director trusts the composer with broad trust, what the composer has to do is to write some scary music that they think really works with the scene, and that they think sounds awesome. They even have a starting point from which to start experimenting, namely the use of lots of percussions. This is the kind of brief that composers love.
However, if the director has narrow trust, then the composer has to write some scary music that they think that the director will think really works with the scene and that they think that the director thinks sounds awesome.
In fact, the fact that the director has specified instrumentation suggests that the director has a pretty clear idea of what they want in their head; unfortunately, all that the composer knows is that it involves percussion.
In this case the bulk of the composer’s work is, through trial and error, to find out what it was the director had in their head in the first place (and by the way, because the director isn’t the one who’s a professional musician, they probably won’t have the technical vocabulary to describe what they want, so much of what the director asks for will not be what they actually want).
Essentially in this scenario, the director trusts that the composer will successfully read their mind.
If you’re charged with delivering a project by your boss, and you want to run it in an Agile way, you’d better hope that your boss trusts you with broad trust. Because each sprint it’s going to be you who decides what to build next, and how to balance the competing requirements of diverse stakeholders.
Your boss is going to be surprised at what they see at each sprint review, and if your boss isn’t happy for you to surprise them, things will not progress smoothly.
So if you lack the sort of trust that lets you actually own your product, what can you do?
You can try essentially sharing the Product Owner role with your boss—give them the fun bits about making decisions around prioritization of tickets, while you do the day-to-day grind that they won’t have time for.
This is not ideal, and won’t work unless your boss is prepared to put in the time to be continuously involved.
The other option is to sigh and abandon an Agile methodology and go back to building a full spec in advance which you can get your boss to sign off on. That way they don’t need to trust you because the spec takes all the decisions for you.