Evolving Scrum with Kanban #2 – Measure What?

In the second in this series of blog posts, I would like to tell you about my journey into metrics; what difficulties I had, what I learned about the use of metrics, and how metrics will help to improve the chance of successul change.

The third Kanban practice tells us to “Measure and Manage Flow”. When I started to use Kanban, I had my dificulties with this. First, I wasn’t sure whether it would be okay to measure performance of my team (hint: it’s about measuring the performance of the system, not the team) and second of all, I just didn’t know where to start. Nonetheless, I gave it a try and was confronted with the question: “What should I measure?”.

Of Metrics and Charts

In learning about the Kanban Method you will stumble over metrics like lead time, cycle time, throughput and flow efficiency. So did I. They are well described in David’s Kanban book. What I didn’t get was, whether these metrics are mandatory or optional, and how they would help me foster change successfully. Not enough, searching for answers to my questions I discovered more stuff I could measure.

With the different kind of metrics come charts like Cumulative Flow Diagram (CFD), Control Chart or Lead Time Distribution Chart. Just to name a few. Please forgive me, I won’t be telling you about the different kind of charts. There are plenty of helpful articles written by others about charts. Check out Pawels Brodzinski great article[1] about Cumulative Flow Diagrams for example.

None of this is rocket-science, but knowing what I should measure (e.g. lead time) I still didn’t know how to measure. How should I measure lead time? Where should I start to measure lead time? Lead time of what?

What should I measure?

If you read the first article of this series, you have seen the Kanban board we used. It consisted of the different knowledge discovery steps, representing the workflow of the developer team and included a Scrum-like task board in the middle, where the tasks of a User Story would be visualized.

The problem I had was not knowing whether I should measure the lead time of User Stories or lead time of tasks belonging to a User Story. What should I measure if User Stories were split halfway through? Should I measure the User Story we had split or should I measure the new User Story originating from the old one?

Take a look through the Kanban Lens

To find the answer to this, let’s remember that the Kanban Method is a

  • method for directly improving service delivery
  • mechanism for catalyzing continuous improvement

And let’s take look through the Kanban Lens[2] to remind us of the perspective Kanban wants us to view work

  • service-oriented
  • service delivery involves workflow
  • Creative work flows through a series of information discovery activities

Kanban is all about service-delivery and improving it. Talking about a service, there has to be at least one customer. If we want to improve our service, we should ask our customer what he cares about.

Know your Customer

In my case, the Product Owner of the team was interested in getting the User Stories that she gave the team to work on, back as soon as possible. She had no interest in how fast tasks would be finished. All she wanted to know was, how long does she have to wait to get a User Story, she asked the team to finish, to be released. This was valuable to her.

For my particular situation, a Scrum team using the Kanban Method on Flight Level 2[3] to improve their service delivery, this was the answer to all my questions.

What should I measure, User Stories or tasks? User Stories!

Where should I start to measure? From the point in time the User Story was put onto the board by the Product Owner, the replenishment, to the point the story was ready to be released!

How should I measure split User Stories? Measure the lead time of original one, and help the Product Owner to avoid such User Stories in the future!

Caution! Pay attention to your altitude

Measurements look different if we move up to Flight Level 3, where we are looking at Minimal Marketable Features (MMF) to be delivered, consisting of many User Stories.

Chances are high that not only one software development team is working on it, but also (online-)marketing, business intelligence and maybe even support are adding value to it as well. On this level, the metric “User Story lead time” loses its value. To measure the lead time of the MMF is obviously more important.

Copy and Paste isn’t going to work

One metric, lead time, two different environments, Flight Level 2 and 3, equals two different measurements. So be careful not to copy metrics from one environment and paste them into another. Always ask yourself these questions:

  • What is the goal I want to achieve?
  • Which metric would support me in achieving that goal?
  • How long do I measure?

Answering these will help you make your measurements more effective and not waste your time. Plus, you will come up with more metrics like:

 More metrics you will find

  • defect rate
  • WIP limit breach
  • time spent on white noise
  • customer satisfaction
  • team member satisfaction
  • the time tickets were blocked
  • time team members do something “real quick”
  • time we wait for external suppliers

These are measurements I used myself or saw used by others. I am sure this list is far from complete. There are no limits to what you can measure. What you measure depends solely on what you want to improve and keep an eye on. (Lead time normally isn’t needed to improve, but a good indicator if something is wrong).


More important than knowing the tools you can use to measure, is knowing what you should measure in your Kanban system. Knowing which metric will help you to improve, and when to use it is a very strong weapon you don’t want to miss out on in a changing environment. It will help you to get a better understanding of the current situation you are in. And sometimes even a Gantt-like Chart can help. ;-)

What are your experiences? Any feedback? Please leave a comment and share your thoughts. Thanks!

[1] http://brodzinski.com/2013/07/cumulative-flow-diagram.html
[2] http://djaa.com/kanban-lens
[3] http://www.leanability.com/kanban-flight-levels/

Leave a Reply

Your email address will not be published. Required fields are marked *