A guide to Zen Agile

… or “Sh*t happens, get over it”

“Agile” is one of the most discussed topics in tech. You might say that we’re in a ‘post-Agile’ era, with it being ‘de rigueur’ for software development.

However, just as much is written about the tendency of senior management to think that Agile is a magic trick; enabling teams to do more with less (and faster), rather than something they themselves need to adopt.

Is agility a higher state attainable only by elite outfits like Spotify, or is it something that any company can achieve?

So, what is this Agile thing anyway?

It’s important to remember that Scrum and Agile are different things – it is perfectly possible to use Scrum without being Agile, and vice versa. So, when Drift CEO David Cancel shouts “I hate Agile,” perhaps what he means is: ‘I hate when people do a Scrum thing and think that means they’re Agile!’ Or put it another way. Agile as an industry of training, consultants, ceremonies, and artifacts doesn’t necessarily translate into businesses becoming agile, nimble, and flexible.

In 2016, a Deloitte analyst inadvertently portrayed this maelstrom of jargon in his infamous ‘tube map’ diagram. This diagram showed (some of) the various tools and practices advocated in the Agile world.

Ugh – easy to see why someone might think Agile has lost its way.

Agile is something you should be, not something you should do, and no tool will make you Agile.

Let me give you some personal examples…

A former CEO I worked for couldn’t get his head around why I’d sometimes reposition half-way through a meeting. His preference was to relentlessly persevere with “the plan,” an always-out-of-reach goal to “arrive in style.” Needless to say, he was a former project manager from whom the word “Agile” literally elicited a wince.

Living with such constraints is an ordeal for Agile thinkers.

As a university student, I was once rushing to catch a train home to Durham at the end of a busy week. I ran into the station and onto what I thought was the 17:25 waiting at platform 2. Imagine my dismay when we departed in the wrong direction, and the guard then confirmed that the first stop was Edinburgh (in 2 hours). My half-hour commute had just turned into a 4 hour round trip to Scotland.

Most of it was spent sitting in an empty carriage, and with the darkness outside, I couldn’t even admire the scenery. Not planning for this impromptu trip, I had nothing to read other than the train company’s brochure, urging me to take a day trip to historic Durham. It’s not unlike working in a project-driven company and reading blogs about A/Agile.

On another occasion (stay with me) when working as a consultant, I rushed out of the office, jumped in the car, and was bombing southward from Newcastle on the A1. As I passed the Scotch Corner services, a sudden bolt of awareness struck cold fear in my spine: Where was I going? Stopping, I got my laptop out to check my diary (no smartphones yet) and confirmed that I should have been heading North to Glenrothes. Fortunately, the unintended detour only added 90 minutes to my journey.

Did you spot the analogy?

The train is Waterfall and the car, Agile. In both cases, I took the wrong initial course. But, on the train journey, I had to continue until the bitter end.

I did ask the guard if he would stop the train to let me off, but he just laughed at me with the incredulity of a PRINCE2 program manager when you ask them to pivot.

For me, being anything other than Agile is effectively to pretend that (a) you can predict the future and (b) sticking to a bad plan is better than changing it.

So why does Waterfall still exist, given its dire reputation in software product development?

Letting go of the future is counter-intuitive

We, humans, are instinctively able to discern patterns in things. We use this to foretell the future by extrapolating from our past experiences. Side effects of this include an emotional need to know what’s going to happen, and a predisposition to confirmation bias. Doing a familiar task makes you feel like you’re on safe ground and you can go a long way along the road to damnation before reality intervenes.

So, when planning a Waterfall project, everyone involved conspires to delude themselves that they’re going to release ‘on time’ and ‘on budget.’ Execs push a date at which the release must happen, and delivery teams make promises based on estimates. Devs complain about this, pointing out that it’s impossible to accurately forecast software development, so someone adds ‘contingency’ to the made-up numbers. Everyone starts out feeling comfortable and positive because there is a plan.

Before long, however, stakeholders start to wonder what’s happening. As there is, as yet, no software to be seen, their nerves start to jangle about the pace of progress. They enquire into how things are going, only to be told that “There’s nothing to see yet, we’re still architecting the platform.”

The lack of visible progress on a Waterfall project can lead to some nervy stakeholders. The anxiety builds to the point where they demand to know whether the deadline is going to be met and that something tangible is produced. The delivery team grumbles about ‘scope creep,’ ‘changing requirements’, and having to make ‘shiny stuff.’ This negativity is driven by a growing feeling that control is slipping from their grasp.

At the same time, they can tell the suspicion is growing from people outside the team who ‘just don’t understand how hard this is.’ The deadline is extended, tempers fray, additional pressure is heaped on the team and people quit. We’ve all seen it before. When running a project this way, you should start out by writing the post-mortem!

In truth, it’s not feasible to accurately predict how a software project will pan out, as the control was never really there in the first place. This is the main reason for the ‘inspect and adapt’ concept in Agile.

It encourages the delivery team to review their progress early and often, thus reducing the risk of doing the wrong thing for a long time. You may have seen the following model for comparing risk between Agile and Waterfall methods.

The idea is that in a Waterfall project, the risk remains high until delivery. Whereas, in Agile, the risk is reduced from an early stage by iterative inspection and adaptation.

However, this analysis of Waterfall is rather optimistic. In practice, it often creates a graceful, slow-motion failure, founded on stated prejudices (AKA “requirements”).

A bulletproof change control procedure will cover the Project Manager, and even make them look good. The aim is not to create value; it’s to ensure that a plan is followed. Boxes ticked, meetings minuted, excuses made. All too often, the outcome is an unrecoverable failure and, having spent so long building the wrong thing, there’s no budget left to rectify it. The first-mover advantage has been lost.

The deadline delusion

It’s Agile blasphemy, but also an inconvenient truth, that deadlines are inevitable in business.

Building a product costs money and there is generally a finite amount of it available. Therefore, stakeholders are understandably keen to spend as little time as possible spending it. Unfortunately, in software projects, deadlines are nearly always missed. This can be because:

• Customers always want stuff yesterday, and your competitor is threatening to give it to them, leading to pressure to say “Yes, soon.”
• Technical people often overestimate their abilities and neglect to account for mistakes, lost time, and unforeseen circumstances.
• Once the team is making something, they feel good and fall into the trap of thinking there is plenty of time left. Only when things are obviously off track does anyone intervene, and by then it’s too late.

In fact, if you meet a deadline, it’s probably because the stakeholders were prepared to overpay to guarantee it.

Transcending the tyranny

How to escape this?

Permit me one more confessional anecdote: As a heavy smoker in my twenties, I resisted giving up for many years. I was unable to countenance a future as a clean-living kill-joy and felt that my life’s course was already set. When I finally decided to quit, it was genuinely terrifying for me because it felt like dying. Based on the smoker/rebel template, my personality was ending, and I didn’t know whether there was life after the death of this self-image. It was a traumatic experience, but I survived (obv), and this major version upgrade taught me some life lessons.

Lesson 1: The only thing that really matters is what happens next
Breaking my self-fulfilling prophecy that I’d always be a smoker was hard. I’d constructed a mental life script for myself, with a sense of inevitability based on my sense of self.

Unfortunately, human intuition, which generally serves us well, also suffers from systemic faults such as the sunk cost fallacy. We tend to persevere with a fruitless course of action because of the cost already incurred. We think, “I’ve had the bad luck, some good is due soon!” Nope, any money you’ve wasted on a product is now in the past, and the only factors governing your future success are happening right now.

Like Airbnb, great products came about as a result of many sprints; with many mistakes and brave decisions made along the way. A stretched analogy perhaps, but life and business are anything but deterministic.

Lesson 2: If you decide to keep going for something, you’ll probably get there, eventually
One day, around a year after quitting smoking, I realized that I was no longer thinking about it all the time and had no inclination to go back. This only happened because I had decided on 17th March 2001 at 13:35 GMT to become a non-smoker. A definite goal, but with an uncertain time scale.

Similarly, my car trip had a destination (despite the detour to Scotch Corner). If there had been no ‘north star’ goal, I might have found myself endlessly negotiating the one-way system in Middlesbrough – keeping very busy but not actually going anywhere. To paraphrase Lewis Carroll, “If you don’t know where you’re going, any road will take you there.”

Agile is not random. Rather, it’s mindful pragmatism that keeps you moving in the right direction. You continually apply course corrections, but always with a destination in mind.
Fans of classical project management will argue that this can be handled via change control. However, such frameworks are predicated on the assumption that change is an exception.

Conversely, you can think of Agile as continuous change control. Although, we shouldn’t talk about “change” and instead recognize and indeed embrace uncertainty.

Throughout my career, I’ve heard delivery teams complain about changing requirements and “scope creep.” In most cases, the client hadn’t really changed their requirements. It was just that the implications of their vague ambitions had become progressively clearer on closer inspection and as time passed.

Lesson 3: Fear causes all the trouble
Channeling that great business philosophy guru, Yoda: “Fear leads to negativity; negativity leads to inertia; inertia leads to poor quarterly results.”

The reality is that it doesn’t matter how much you plan; things aren’t really under your control. There’s no point in being afraid.

What’s the worst that can happen? Well, you could lose your job? Faced with this worst-case scenario, it’s worth asking yourself; “If I’m miserable here because of constrained thinking, lack of trust, and systemic fear, shouldn’t I go somewhere else?”

However, it is disruptive to have to keep switching jobs. So, how can you overcome the deadline delusions and control-freakery?

The trick is to understand that stakeholders are also prone to fear, but their confidence may develop over the lifecycle of a project (see chart below). They usually don’t grasp Agile and expect to see a linear progression, such as you might see with a house taking shape on a building site. However, making a new product is far less predictable than making a building, and it’s important to understand how stakeholder confidence changes over time:

In the Waterfall case, confidence will be either restored or shattered completely at release time, depending on whether the product meets the expectations that have built up over the project.

In the Agile case, early confidence can be dented as incomplete stages of the product are produced by the delivery team. But, so long as the team keeps taking feedback, confidence builds as subsequent iterations move visibly closer to a market-ready product.

A plan is not a goal – it’s a tool, and an adjustable one at that

If you’re building a business and need some software made, you will probably have some idea of what you want your product to be. You will also need assurances from your product team that they can build it in a reasonable timescale, without busting your budget.

So, how to fix the three corners of the scope/resource/time triangle?

Traditionally, this would be achieved by spending a lot of time analyzing the requirements in detail, planning exactly what’s going to be built, estimating it, then adding contingency and a change management process. If a high degree of confidence in the time and cost is required, the price is often accepting a much longer timescale and a higher risk that the eventual solution will fail to match requirements that emerge or change during the project.

Agile is often seen as a silver bullet. Don’t plan beyond one sprint! Pivot every now and again! Prioritize everything, brutally! Que sera sera! Zealots will shout down any talk of roadmaps, planning, or strategy.

Of course, I’m being a bit silly and nobody would take this seriously as a business approach (you would think). Businesses happen because someone had a vision and pursued it with vigor and persistence in the face of adversity. It is the founder or business leader who takes a punt on and is accountable for an idea. Agile is great for handling a change of direction in the detail or even means of execution. But, if the big idea fails, the business is gone.

Relax, nothing is under control

To sum up with a Zen-like platitude…

Beware the duality trap, in which one believes they can plan the future of a product.

Learn to accept the reality of the current sprint; appreciate its innate business value.

And, be comforted by your career prospects, which will transcend the noise of ephemeral shouty bosses if you keep adding value.

Author’s bio
Aidan is a Product Focus Senior Consultant.

He has held Head of Product, Product Director, and Chief Product Officer roles in technology vendors and digital agencies, within a broad range of industries.

Aidan has in-depth experience of the full software lifecycle, including start-ups, scale-ups, and mature product portfolio management.

Read more about Agile.