A friend and colleague posed the following question via a text message:
What are the leading indicators that a program increment or team will not deliver on their commitment? What metrics would you gather to prove them out?
Before I dive into my thoughts around this, let’s setup a couple of assumptions. First, let’s assume we have a Scrum team that is following the Rules of Scrum. But along side that, let’s also assume we have a team that follows the spirit of the Agile Manifesto. I talk more about how I stand up a new team and my ideal relationship between the Agile Mainifesto and Scrum here.
With a Scrum Team, their commitment to delivery is to their Sprint Goal. While they forecast the stories they will work on and try to complete, they still commit to delivering their Sprint Goal.
When a Scrum Team works in a Program Increment (most likely as part of an Agile Release Train in a Scaled Agile Framework implementation), the team also commits to delivering Program Increment Objectives, or PI Goals, statements that summarize the value they intend to deliver.
So how do we measure if a team is on track toward delivering Sprint Goals and larger Program Increment (PI) Goals?
Is a team on Track to meet their Sprint Goal?
There are a few indicators we can use to help us understand how a team is progressing towards their sprint goal. However, none of these are perfect. All that they allow us to do is to ask better questions.
Burndown (or Burnup) Chart
Burndown or Burnup charts have long been a powerful tool for agile teams to inspect how they are tracking towards completing the sprint. A typical Burndown or Burnup chart tracks how the team is progressing towards completing all of their stories within a sprint. If the team is on pace to complete all of their stories, that is a good indicator of success. Conversely, if the team is not on pace, that may indicate a problem.
However, there are 2 problems with Burndown/Burnup charts. First, despite efforts to complete work early and often in a sprint, many teams tend to complete more stories the last couple of days in the sprint. So, a team that appears to be off track might surprise you and complete everything.
Second, and perhaps more importantly, the Burndown/Burnup chart does not track completion towards the Sprint Goal, just the items that make up the sprint. The Sprint Goal might entail completion of 1 or all of the stories in a sprint, or somewhere in-between. So a team that completes just 1 story might indeed achieve their Sprint Goal, if that story was all that was necessary to accomplish the goal.
Injected Work
$#!& happens. An emergency issue occurs in production. A user story was missed. An important bug needs to be addressed now. Whatever the case, a team that operates in an agile fashion has to respond to change and bring in work into the sprint. We call this “injected work”.
Focus is one of the 5 driving values of Scrum. Injected work will force a team to lose its focus, which can jeopardize the Sprint Goal, the stated focus for the sprint. While tracking Injected work does not correlate directly to a team completing their goals, it can indicate trouble lies ahead.
A metric of a bygone age may also be useful here: defect rates. In a world of waterfall projects, Project Managers would track defect rates as an indication of progress towards completing a release. The idea being that once the level of defects found level off and no more critical defects exist, a project is ready to release. For Agile teams, tracking defect rates may also be useful, as this indicates quality of the product. Products with poor quality will likely have larger amounts of injected work, have higher technical debt, and cause teams to run into more issues as they progress towards their Sprint Goal.
Is a team on Track to meet their PI Goals?
Many of the metrics for a Sprint Goal can also be applied towards larger PI Goals. Burndown charts can be particularly useful, when the team identifies the critical path.
Track the Critical Path
Not all stories that a team plans in a Program Increment are critical to the success of the PI Goals. Some of them are simply “things the team needs to do”, like update code libraries, adhere to a new legal requirement, or address technical debt. But, there are other stories that are crucial to the success of a PI Goal, and these stories lay out a Critcal Path to the teams success. If we ask the team to identify this path for us, we can track it via a burndown chart and see their progress towards the end result.
Like any other metric we’ve discussed so far, this isn’t perfect. It’s possible for the team to identify new stories, causing the critical path to change. Depending on the goal, it’s also possible for the critical path to be completed, only to find out that we made incorrect assumptions along the way.
Actual Velocity vs Planned Velocity
When planning a Program Increment, the team plans stories based on their Planned Velocity, or the velocity they expect to achieve based on what they’ve accomplished in the past. By tracking the velocity the team planned to achieve vs their actual velocity, we can see if the team is accomplishing more or less than they anticipated.
Of course, by now you’re expecting me to state a negative to this approach, and this time it’s a big one. Just because a team is accomplishing a lot of work, it doesn’t mean it is the right work to achieve the Sprint or PI goal. It could be made up of injected work, technical debt, or simply stories that do not align with the team’s goals.
One Radical Idea
As we’ve examined a number of metrics, there have been issues that arise with each of them. So what is a team, train, or leader to do to tell if a commited goal is still achievable, in jeopardy, or out of reach? One radical idea is also the simplest: ask the team. In SAFe, we establish commitment to PI goals based on a confidence vote from the team. The team are the ones committing to the work and the ones closest to it. Thus, they should be in the best place to gauge whether the goal is on track or in jeopardy.
One Scrum Master that I work with asks the team for how confident they feel they can achieve the sprint goal at the end of a stand-up midway through the sprint. The team votes 1 through 5, and when a team member votes low, she asks what can be done to increase the teams confidence in reaching their goal. This is self-organizing teams in action!
Concluding Thoughts
We’ve discussed a number of different metrics. Each of them have their place, and can help you understand how a team is progressing toward their goals. However, all that these metrics do is allow a team, train, or leader to ask better questions. A team may be tracking perfectly towards their goal, but one missed story could cause the goal to be missed. Conversely, a team could have horrible defect rates, lots of injected work, and a poor velocity, yet still achieve their goal.
Ultimately, the team doing the work will have an idea of whether the goal is achievable or not. Teams tend to be overconfident, but that is where I see the value of metrics. Bring in metrics to challenge the teams confidence, but when they are still confident, leave them to the task of completing the goal. And if their confidence was misplaced, let’s ask why we were overconfident, so that next time, we can do better in achieving our goals.