Scrum: what it is and how it works

As the most popular and widely used Agile development approach, it has become synonymous with Agile software development. It does not dictate how everything should be done on a project but leaves many details to the development team. However, it does have a set of defined roles, rituals, terminology, and tools. The following section describes a classic Scrum implementation.

Scrum explained

Scrum is named after the close-knit shoulder-to-shoulder formation used in the sport of rugby.

Scrum is based on a series of short and focused development projects called sprints. These are time-boxed, which means they must deliver by a set date. The deliverable from each sprint should ideally be something that a customer could use (known as a potentially shippable product increment).

Roles

There are 3 defined roles in the delivery of Scrum projects, the Product Owner, the Scrum Master, and the team.

The Product Owner is the customer proxy and is responsible for ensuring that what the development team builds delivers value to customers. Fundamental to their role is a good understanding of customers and their needs. With this insight, they select, prioritize, and communicate requirements, often written as ‘stories’, with the development team. The prioritized list of stories is called the product backlog, and the process of adjusting priorities and breaking them down is sometimes known as backlog refinement. Product Owners normally sign-off what has been built to confirm that it is complete.

The Scrum Master champions and facilitates the Scrum process, coaching the team and tackling any impediments (i.e., blocking issues) that get in the way of delivery.

A team is a cross-functional group of, typically, 4-10 people who build software (effort estimation, analysis, design, development, testing, documentation, and progress tracking).

Process

At the heart of the process is the sprint. This is a set block of time, usually from 2-4 weeks, where the team builds the requirements agreed with the Product Owner.

The sprint planning meeting happens immediately before a sprint. It finalizes the list of requirements to be built. In the meeting, the team will discuss the potential requirements, refine their view on the effort required to do them, and agree on what will be delivered.

 

During each sprint, the Scrum Master will run a daily stand-up. This is a short, e.g., 15-minute review meeting to track what’s going on, and takes place face-to-face or remotely. Attendees answer the questions. What have you done since the last meeting? What will you do between now and the next meeting? And Is there anything that will stop you from doing what you have planned? If issues are raised, then it’s the Scrum Master’s job to take those issues offline (rather than wasting time during the stand-up) and ensure they are resolved.

It is increasingly common to run a backlog refinement meeting halfway through a sprint in preparation for the next sprint. This meeting allows the team to discuss and estimate requirements from the prioritized backlog. Input from the team is key in aligning priorities. It also helps the Product Owner identify gaps in customer insight and major ambiguities in the requirements, so they can be resolved in readiness for the next sprint planning meeting.

Once the sprint is finished, a sprint demo is organized to show the various stakeholders what has been delivered, to get sign-off from the Product Owner, and to provide the team with direct feedback.

Sign-off from the Product Owner means that the requirement is ‘done’. It’s important that the definition of ‘done’ is clear and agreed upon. As a minimum, it will mean that what has been built has passed its acceptance tests and that it incorporates ‘standard’ requirements (e.g., help screens, alignment to product architecture and design). A broader definition might require that release notes and user documentation have been created, that the release is packaged and ready to go live, and a record made of any technical debt created.

Terminology confusion Scrum comes with its own terminology, and it’s easy to assume that everyone means the same thing when they mention terms such as stand-ups or stories. If you’re new, it’s always worth checking. One client recently talked about using ‘stories’ to describe requirements, and our assumption was that they were using ‘user stories’ or ‘job stories.' In fact, this Agile organization was using classically written requirements, i.e., ‘The product shall do XYZ.' In another organization, while talking to the Chief Product Officer, we asked about the Product Owners’ responsibilities in Agile only to be told that they weren’t Agile but were wholly Waterfall in their approach to development!

Following this, a sprint retrospective meeting is held to review how things went, whether the forecasts for work completion were accurate, and to identify any other issues. The focus of this meeting is on learning, so the next sprint is better (stakeholders outside the core team are not invited). It’s an opportunity to put the Agile mindset into practice with experiments on ideas for improvement, for example, running a workshop to better understand a problem area.

Prioritization techniques

There are various considerations when prioritizing in the sprint planning or backlog refinement meeting. These include the goal for the increment to the product, the value to customers of new functionality, user experience improvements, alignment to strategy, requirements interdependencies, or reducing current or imminent technical debt.

A common approach is to evaluate requirements based on their value (to the organization) vs. cost and urgency. One example is Weighted Short Job First (WSJF), described in the table below.

Weighted Short Job First (WSJF) This prioritization technique prioritizes the delivery of high-value requirements. It uses two attributes of each requirement: Value and Duration (to build). Value will often be measured by a Cost of Delay or a Fibonacci number. Duration is measured in days or weeks of effort or a Fibonacci number. Dividing Value by Duration gives a priority order with the highest scoring items done first.

Common variations to the ‘standard’

While this article shows the standard way of working, there are many variations made to ensure it works in a particular business.

For example, while purists argue the team should be self-organized, in practice, we’ve seen the need for experienced members to provide insight into estimation and work allocation.

Planning Poker A challenge with prioritization is the estimation of effort and value. It can be tough to estimate value in monetary terms, and it can be tough to estimate in working-days of development effort. As an alternative, value and effort for each story are often measured relative to each other using a somewhat abstract metric such as t-shirt sizes e.g., Extra small, Small, Medium, Large, Extra Large, or points e.g., 13 points of effort. With points of effort, to avoid endless debates over whether a story is estimated at one of two close numbers e.g., 17 or 18 points, the team are asked to choose an estimate from a Playing poker Fibonacci sequence (1, 2, 3, 5, 8, 13, etc.) where there are fewer big numbers. All estimates are revealed at the same time (so as not to influence each other). Significant variations are discussed and a consensus agreed. Any requirement that is given a very high score (e.g., that’s likely to take longer than half the period of the sprint to deliver) will be broken down further and each part estimated and prioritized. This technique is called Planning Poker.

Another challenge is that while each development team is supposed to be multi-disciplinary, we often find that specialist skills are needed e.g., on databases, on user interfaces, or on security. We see two common solutions to this. The first is to schedule these specialist requirements into the sprint team that has the expertise to deliver. The second is for an expert to be brought into an existing team for a period of time to do what’s needed. Either approach creates the challenge of having to plan time with the specialist that has the right skills and experience.

What is DevOps? Tight interworking between the people in Development and Operations groups to enable the faster release of newly built software into production. It is facilitated by having aligned development and operational environments, automated testing, near real-time performance monitoring, and other techniques.

While the result of each sprint might be a ‘potentially’ shippable product, what often happens differs from this. The sprint itself might be an ‘investigative spike’ where the development team explores technical approaches to building something rather than doing the build. The team might be operating in a DevOps continual delivery mode where code is automatically tested and, if the business wishes, released directly onto an operational platform. A third common outcome is that what’s been built is insufficient value to customers to justify either the effort of launch or for customers to upgrade. In this case, multiple sprints will be used to add more functionality, ending with a hardening sprint that performance tests the product prior to launch. This sequence of sprints delivers a higher value product to customers.

The Product Owner and Scrum

A key role for the Product Owner is to present the market priorities for each sprint. This is done by constantly managing the product backlog (see diagram below) and prioritizing based on customer insight, development estimates, and alignment to business priorities.

What finally gets agreed for inclusion in the sprint is often a combination of these requirements, input from the technical team on what they must get done (e.g., tech debt, interdependencies, architectural changes), and input from the user experience team.

Writing requirements

Requirements are documented as stories (typically user stories or job stories) at varying degrees of detail depending on their priority. To start with, requirements are normally documented at a high level, termed an Epic. This is a large requirement that describes broad functional needs.

In preparation for the sprint planning meeting the highest priority requirements, those to be built in the next sprint or two that are part of an epic are broken down into smaller requirements (i.e., those that could be built with just a few days’ efforts). The format of requirements is usually that of a user story or job story.

User stories are used where the requirement is to deliver functionality that lets someone do something, and the format is a short, simple description written from the perspective of that person. A typical format is:

As a <type of user>, I want to <some goal> so that <some reason>.

For example: As a user of email, I want to change my password so that I can maintain the security of my account.

Job stories are used when a piece of functionality is triggered by a system event. A typical format is:

When <trigger / situation> I want to <motivations / forces> so I can <expected outcome>.

For example, When a mail attachment size exceeds a system threshold level, then I want to record details of the attachment size and format them into a log, so I can get a report of when and how often it happens.

Whatever format is used, acceptance criteria also form part of the story. Often these are started by the Product Owner and refined with the team.

Stories are described in more detail in our Requirements Product Management Journal.

Tracking and estimation

One of the most common things used in Scrum to manage and track the status of work in a Sprint is a task board.

Typically, a row is allocated to each user story with columns representing To do, In development, In test, and Done.

As we’ve already described, estimation in Scrum is often simplified by only estimating development tasks relative to each other, and this is done by awarding points for the size of each task. As each sprint is the same length of time, the team learns how many total points can be delivered in a single sprint. This is known as the team’s velocity. The velocity of development is tracked using a burndown chart. A burndown chart records user stories completed and shows the team if they are on track to deliver the volume of work previously forecast. What tends to happen is that velocity will be slow in a new team working on a new product domain. As they build familiarity, their velocity will improve (as will their ability to estimate reliably).

Great responsibility

The Product Owner sets the development priorities through the product backlog and can change their mind on these priorities at any time up to the start of the sprint when they are to be built.

This gives the flexibility to respond to market and competitor moves or as the understanding of what’s needed evolves. Which is great. However, it also means there is a heavy responsibility on the Product Owner to get it right, which they can only do if they spend the time to get the market and customer insights needed.

It’s easy for Product Owners to fall into the trap of ‘feeding the development machine’. That means focusing only on writing requirements to make sure the development team is kept busy and so everyone can point to how much has been delivered. The team becomes a ‘feature factory’ where success is measured by the number of features built, with little regard to the value of what has been created. The solution is to make sure there is a focus on the business outcome of what’s being delivered (e.g., customer retention, customer satisfaction, or product sales).

Product Owners and Product Managers

Many Product Managers do the Product Owner role. Lots of Product Owner roles expand to take on product management activities. And in other businesses, there are Product Managers and Product Owners working closely together.

However, if you take the Scrum definition, a Product Owner is the role you play as part of the team that is developing the product. And, it is generally accepted that a Product Manager job is a much wider role, one that is responsible for the overall success of the product. We explore who does what in our article Focus – Survive and prosper with Agile.

Read more about Agile.