Developing Metrics for Development

Last month I wrote about my intention to try Cycle Time as a metric for software development. Back then it was still hypothetical, now it's real. Maybe it's still a bit early to draw any final conclusions, but it's already evident that I failed miserably in communicating the reasoning behind the change. And the criticism is well deserved.

I selected the average Cycle Time during past month as the metric for each team. In a way it is a good metric and gives indication about how quickly the value was realized after the work on a backlog item was started. But all backlog items aren't equal and also work type and intensity differ during the release cycle. In the beginning of the release cycle teams mostly work on the new development. Nearing the end the period emphasis is shifted more towards testing and bug fixing. Issue life cycle tends to be much shorter with bugs than when creating new functionality.

It is also questionable if one monthly average number can help in decision making, in essence serve as an actionable metric. You can see the trends in longer time intervals, not much more than that. Another angle is that a team can be really mature and produce outstanding results even though they don't split their Stories into very small parts. Their average deviation can be small and thus predictability high.

Then again I feel that the metric got a little unfair criticism when it was compared with Lead Time. In the end the company is interested in how the customers experience our products and services (Customer Experience = CX). But this includes many topics that are out of scope for software development. Like the company brand(s), how salesmen conduct their business and how the customers are served by the customer service. I think the software development part of the system (company) can greatly affect the User Experience (UX).

 

I would love to get the big picture correct. So personally I feel that the CX is more important than UX alone and Lead Time more important than Cycle Time. But in both cases the internal part is relevant component of the whole and deserves a metric of it's own. (It would be so cool if I could somehow bundle measuring UX and Cycle Time somehow... Probably there's no such easy connection. I think relationship between Lead Time and Customer Experience is much easier to show.)

Now the current bleeding edge is to try using the whole information provided by the JIRA Control Chart instead of simply one number. I haven't yet figured any cons, but on the pro side you get trend lines of where the performance is developing and standard deviations. Of course you can still identify the release cycle from this data, but you can draw other conclusions too. Increases in rolling average or standard deviation can signal trouble. And when the figures are followed regularly, the team can be aided in a timely fashion.

 

Currently I feel like I'm making too rushed decisions. I'd like to involve other people more and ask for their input already before I make an experiment. Now I mostly just measure the impact and make amendments, but I'd like to concentrate on this more. Let's see if the Santa Claus and New Year will bring a difference. In case this will be my last blog post this year, I already wish you all Merry Christmas and Happy New Year!

From Forecasting to Cycle Time

I like measuring for the sake of improvement. I believe that measuring something helps to understand its nature. And by measuring you might learn if something is good or bad and going to positive or negative direction.

We have been using so called forecast hit-rate to measure how well the Scrum Teams are able to predict their velocity. It's the result of dividing planned number of Story Points with the actually realized number of Story Points for a Sprint. But in case the number of stories is small, there's easily large deviations when things don't perfectly as planned. And usually things don't go perfectly as planned.

Also this measure can result in some rather nasty dysfunctions. Teams start to favor keeping the forecast instead of trying to maximize the amount of customer value in a Sprint. Or even though some Story becomes obsolete it isn't dropped because that would flaw the statistics. This isn't nice.

We use Atlassian JIRA and JIRA Agile for our Backlogs and Scrum boards. (I guess I should start calling it JIRA Software soon.) It offers a handy Velocity Chart where one can follow these planned vs. completed results. This is essentially the source of our forecast hit-rate information.

But due to all the possible events that affect the Scrum Teams and the #NoEstimates movement that gains momentum also in our company, the data has a really high variance. Even after couple of years it shows no signs of stabilizing. (Of course it can argued that is a systemic dysfunction. As it is. But let's not go there.)

Another tempting alternative for this metric that I've been thinking about is Cycle Time. Average time that a Backlog item takes between the start of work until it is completed. Let's assume that such metric would be easily available and that there would be target on it or that management would follow the number. What kind of behavior would this drive?

My assumption is that it would make the Stories smaller. That teams would strive to get things done faster instead of having mammoth size stories. So In the end there would be a bigger number of smaller Stories and tasks that could be made in shorter time. I would like that. Also it would fit nicely with #NoEstimates.

Since this is still simply hypothetical, I don't have any results yet. But this is an experiment that I'm about to try.

Focusing is Effective

If you have a hypothesis, there's probably no better way to validate it than by testing. It's one of my favorite practices from Lean Startup. Experiment. Make a hypothesis, test it and learn. Repeat.

But you also need metrics. Something that you can measure and see if your actions have impact on them. In our case we have been asking our Developers, Product Owners and Product Managers the same set of questions twice. (The poll has been anonymous and people have been participating really well.) The results show nicely how the actions we have done (our focus) have improved the results in those areas. Unfortunately they also show that things that have not been paid too much attention to are not improving.

We have made big efforts to improve our Product Management and to shed more light to our Product Portfolio. Also we have invested in training our Product Owners. The results in these areas are almost stunning. According to POs and PMs the work in Portfolio has improved by 30-40%! I could call that from zero to hero. Developers are now more satisfied with their Product Owners. Although the sample size with Developers is lot bigger than with the other two group, the numbers have gone up by almost 14%.

But unfortunately there are two sides in a coin. While we have put a lot of emphasis on the Portfolio and on Product Owners, we haven't maybe paid enough attention to core of the agile. Data is ruthless and clearly shows that our efforts in Continuous Improvement and Visibility to Work have not been too good. For this I will take personal responsibility. I'm currently both Chief Scrum Master and Manager of Releases. I have been clearly concentrating too much on one part and neglecting my other responsibilities for the organizational improvement efforts and making things visible. Point taken, now I know what to improve next.

As a statistical experiment this has been very interesting. My interpretation of the results is that they validate our hypothesis. The areas we concentrate on are affected and in a positive way. Having focus does pay off!