Developing Metrics for Development

Last month I wrote about my intention to try Cycle Time as a metric for software development. Back then it was still hypothetical, now it's real. Maybe it's still a bit early to draw any final conclusions, but it's already evident that I failed miserably in communicating the reasoning behind the change. And the criticism is well deserved.

I selected the average Cycle Time during past month as the metric for each team. In a way it is a good metric and gives indication about how quickly the value was realized after the work on a backlog item was started. But all backlog items aren't equal and also work type and intensity differ during the release cycle. In the beginning of the release cycle teams mostly work on the new development. Nearing the end the period emphasis is shifted more towards testing and bug fixing. Issue life cycle tends to be much shorter with bugs than when creating new functionality.

It is also questionable if one monthly average number can help in decision making, in essence serve as an actionable metric. You can see the trends in longer time intervals, not much more than that. Another angle is that a team can be really mature and produce outstanding results even though they don't split their Stories into very small parts. Their average deviation can be small and thus predictability high.

Then again I feel that the metric got a little unfair criticism when it was compared with Lead Time. In the end the company is interested in how the customers experience our products and services (Customer Experience = CX). But this includes many topics that are out of scope for software development. Like the company brand(s), how salesmen conduct their business and how the customers are served by the customer service. I think the software development part of the system (company) can greatly affect the User Experience (UX).

 

I would love to get the big picture correct. So personally I feel that the CX is more important than UX alone and Lead Time more important than Cycle Time. But in both cases the internal part is relevant component of the whole and deserves a metric of it's own. (It would be so cool if I could somehow bundle measuring UX and Cycle Time somehow... Probably there's no such easy connection. I think relationship between Lead Time and Customer Experience is much easier to show.)

Now the current bleeding edge is to try using the whole information provided by the JIRA Control Chart instead of simply one number. I haven't yet figured any cons, but on the pro side you get trend lines of where the performance is developing and standard deviations. Of course you can still identify the release cycle from this data, but you can draw other conclusions too. Increases in rolling average or standard deviation can signal trouble. And when the figures are followed regularly, the team can be aided in a timely fashion.

 

Currently I feel like I'm making too rushed decisions. I'd like to involve other people more and ask for their input already before I make an experiment. Now I mostly just measure the impact and make amendments, but I'd like to concentrate on this more. Let's see if the Santa Claus and New Year will bring a difference. In case this will be my last blog post this year, I already wish you all Merry Christmas and Happy New Year!

Release Planning With Customer Focus

I've written about our Release Plannings already (here, here, here and here), but maybe there's still something to add. These blog posts also help me understand how things have evolved over time.

This time we had agreed the Release Planning Day, our common company event, would only serve as a deadline for having the Release Plans done. So all the actual planning was done before hand. One tool that we extensively used was Microsoft Yammer. It has proven to be handy for sharing something get insights from different stakeholder around the globe. In addition there was a lot of face to face planning between the Product Managers, Product Owners and Development teams.

I had split the day's agenda into the following parts:

• Agenda and purpose of the day delivered by Release Train Engineer
• Where We Are Heading, market insight and Portfolio funnel status by Chief Portfolio Officer
• Product Roadmaps by Product managers
• Lunch and team break
• Each teams' finalized Release Plan by Product Owners
• Launch Plan for each product/solution by Product Managers

I think out of these probably all other items are familiar from SAFe except the Launch Plan. And actually even that is in SAFe. There the name is simply Release. But in our company lingo we have used the word Release for both the Potentially Shippable Increment (PSI) and for the actual Release.

Previously our Release Process ended when the PSI was done. Development and releasing were decoupled. Although this is often beneficial, it had proven to generate a lot of confusion. Sometimes we had a PSI, but no plan on what to do with it. We had concentrated too much on the technical side and forgotten what our whole system was supposed to do: provide our customers with new increments of working software and added value.

Launch Plan is an attempt to look at the same topic, Release, from the customer angle. In bare minimum I want the Launch plan to answer these questions:

• Who are the target audience for this Release?
• How will we communicate about this Release to our target audience?
• How will we deliver the Release to our customers?

Also the Launch Plans seemed to reveal interesting new things about the organization. What we talk about as Release is usually just release of the software component. Creating the PSI doesn't take into account how it will be delivered, configured, trained to new users or marketed. All these are really relevant topics and especially relevant for the customers.

I think generally the Release Planning Day was successful. It was shorter, more focused and took the customer view better into account. In short I would claim it was the best release planning we have ever done. But just as a note for if you plan to try this in your organization: this wasn't our first time. Without the shared history and previous steps on our path I don't think it would have gone like this. So don't try this at home. (Or who am I to decide. Maybe this is the killer recipe that works as a silver bullet. I've tried it once and it worked for me. :) )

As a main takeaway for next time I think we'll be increasing the cross functional collaboration and trying to take all relevant functions into the game, not just development. I think by tweaking our system we can reach so much further than where we are today!

Complexity of Multiple Development Locations

There are multiple reasons why companies spread research and development activities into multiple countries. In some smaller countries it might even be that the job market is not big enough to cover all the needs or that at least talent is really hard to find. That's why it is many times natural to spread activities to locations where the talent pools are bigger. Also the salary level can be a tempting factor especially in cost competitive countries, like India.

But there are certain challenges that are good to keep in mind before taking the step. I have no hidden agendas, my intention is to simply state some things that you might want to consider before starting activities in a new location.

Time-zone differences are good to take into account. Of course they can even be a positive thing if you are looking for 24/7 response times in services and you need to follow the sun. But if you are attempting to have distributed teams (members in more than one time-zone), they will have a limited number of common hours. There are some 'sharing the pain' approaches to this, but none the less it's a real challenge.

I think that if at all possible, people who work together should meet face to face at least in the beginning of their common journey. That helps to build trust and makes the team work easier. It's also easier to explain things and build understanding while being physically close. While this is nice, it will of course create some expenses when people need to travel. Again by far no show stopper but something to consider.

Having multiple development sites has an effect on the communication also. With one site you can rely mostly on physical boards and getting people together, but with multiple sites you need to have proper technology. Things like online backlog tools and communication software become a must. And I think many will find chat tools like Flowdock or Slack beneficial. Internet connection speed will also prove to be important. This depends also on your products, but if you need to transfer gigabytes of data there's a big performance penalty if moving files takes hours rather than minutes or seconds. Thus the local infrastructure also plays a large role.

Job rotation pace also varies between countries. Somewhere it might be possible to stay with one employer for the whole career (maybe not realistic nowadays anymore) whilst in some countries people tend to switch jobs every couple of years. If employee turnaround is really rapid, tens of percents annually, building silent knowledge might be difficult. I mean such information that slowly accumulates while you learn more and more about your field. I think rapid job rotation is also where bad for good team spirit. It's not easy to build trust between people who often change.

Cultural differences should be also taken into account. Countries differ in many things. For example Finland, USA and India are really different. To better understand one another it is good to study a bit what kind of things are valued in the other culture. One nice site to check the differences between nations is here. There you can check how the six pre-selected factors differ. It might explain why your colleagues behave as they do.

Inflation rates are also really different between nations. For example in Finland the tech salaries are high, but there's hardly any inflation. In India it's vice versa. Salaries are in general smaller, but they are raised in bigger chunks and more often. And one of my guidelines is that in the global economy talent is cheap nowhere.

Final thing that comes to my mind is the local laws and amount of bureaucracy. I guess many times the amount of bureaucracy can be bundled with the amount of hierarchy, but that's just my assumption. (What I mean is that if you always need to get bosses bosses boss's signature, you are probably in a hierarchical and bureaucratic setup.) Having a local representative who knows the culture while starting activities up is probably priceless.

Well that was my not so short list of things to consider. Topics are not in any specific or prioritized order. But hopefully they will be useful for people who are pondering about spreading activities to new locations. Make sure you understand the challenges. Some are just risks that may or may not realize, but some of these you will encounter for sure. But if your plan is clear and you are doing it for the right reasons, you have a good change to succeed.

By the way, if you are academically interested in the topic, you might want to check out DD-SCALE project.

SLUSH 2015

If you are not familiar with SLUSH, it's a sort of a startup event. Or I guess at least originally the idea was to gather together startups and investors. And it's still a lot about that, but it has expanded into something huge. Not only startups are interested in being there, but also many established not-anymore-startups like NOKIA, Samsung, GE or F-Secure. Also the set of speakers includes people like former president and Nobel price winner Martti Ahtisaari, president of Estonia and prince of Sweden.

The size of the ~2 day event was around 15k attendees. I included the approximation sign, because two days is far from the whole truth. During, before and after SLUSH the Helsinki region is filled with all sorts of side-events. So it's actually a really great situation to have a product launch. I think at least Comptel, Nokia and F-Secure took advantage of the situation.

I participated in SLUSH as a representative of my company. We were not looking to sell the company, nor to acquire any others. But we are included in the Merit Project along with other Finnish maritime companies. So we had an own stand alongside Eniram, Arctech, Ixonos, Aeromon and others. Honestly there weren't many potential customers, but the emphasis was in trying to suck the most out of the inspiring atmosphere. :)

In the beginning everything was rather shocking. Dim lights and huge laser beams crossing the ceiling. So many companies represented that one got disoriented almost right after stepping in. And due to the fact that I represented my company at the stand and because the event all in all was huge, I didn't watch many of the presentations. But I can recapture some that were most memorable to me.

Risto Siilasmaa told about NGP, Nokia Growth Partners. It's a 700M$ venture capital. Now I have to admit that I'm not that familiar with venture capital companies, but my impression was that they most give 'just money'. Interesting in this NGP concept was that in addition to the investment they also offer a lot of their own contacts, experience and wisdom to the startups. That might be a huge advantage compared to 'just money'.

Then I watched Dyan Finkhousen from GE talk about Open Innovation. I think it was Lean Startup where I read about GE's approach to having separate units that are fully autonomous. They can operate in the new environments without the inertia of the big company. But on the other hand they can also benefit from the 'big shoulders'. I think that's really interesting. There are other similar examples about these sort of intrapreneurs. From Finnish companies Reaktor Ventures is operating similarly.

Ilkka Paananen told that the bar of success has raised significantly during the last five years. Before Angry Birds it was huge to get million downloads. Now with hundreds of millions or billions of downloads it's hardly worth mentioning. He also got big applause for saying that he believes the next Google could come from Finland.

I'm not sure if that is likely, but the atmosphere of SLUSH was such that anything could be expected to happen. I guess many startups could find their investors from the event, but for me it was a source of great inspiration. You are allowed to think big, currently there's venture capital for good ideas and there are no limits. Digitalization is regardless going to happen and it's going to change everything.

(In addition to following the official program I did get some low-quality selfies with various people I admire. One I missed... Daniel, the Prince of Sweden, maybe next time. :) )

From Forecasting to Cycle Time

I like measuring for the sake of improvement. I believe that measuring something helps to understand its nature. And by measuring you might learn if something is good or bad and going to positive or negative direction.

We have been using so called forecast hit-rate to measure how well the Scrum Teams are able to predict their velocity. It's the result of dividing planned number of Story Points with the actually realized number of Story Points for a Sprint. But in case the number of stories is small, there's easily large deviations when things don't perfectly as planned. And usually things don't go perfectly as planned.

Also this measure can result in some rather nasty dysfunctions. Teams start to favor keeping the forecast instead of trying to maximize the amount of customer value in a Sprint. Or even though some Story becomes obsolete it isn't dropped because that would flaw the statistics. This isn't nice.

We use Atlassian JIRA and JIRA Agile for our Backlogs and Scrum boards. (I guess I should start calling it JIRA Software soon.) It offers a handy Velocity Chart where one can follow these planned vs. completed results. This is essentially the source of our forecast hit-rate information.

But due to all the possible events that affect the Scrum Teams and the #NoEstimates movement that gains momentum also in our company, the data has a really high variance. Even after couple of years it shows no signs of stabilizing. (Of course it can argued that is a systemic dysfunction. As it is. But let's not go there.)

Another tempting alternative for this metric that I've been thinking about is Cycle Time. Average time that a Backlog item takes between the start of work until it is completed. Let's assume that such metric would be easily available and that there would be target on it or that management would follow the number. What kind of behavior would this drive?

My assumption is that it would make the Stories smaller. That teams would strive to get things done faster instead of having mammoth size stories. So In the end there would be a bigger number of smaller Stories and tasks that could be made in shorter time. I would like that. Also it would fit nicely with #NoEstimates.

Since this is still simply hypothetical, I don't have any results yet. But this is an experiment that I'm about to try.

Jogging Meeting

I have an impression that walking meetings are currently all the hype. Because everything should be always tested before judging, I decided to give it a spin. But why settle for a walk if you generally move faster?

So we decided to go for a jogging meeting with one of my team mates. I know he's an active runner and I too occasionally enjoy going for a run. And we have one-on-one meetings biweekly anyway.

When clock hit 2 PM we went to staff changing room and took the elevator down. Fortunately my team mate had been smarter than me and thought of a good route before hand. I had just thought about it, but didn't ever have time to do that. As one can guess, route length is an essential factor for keeping the meeting in schedule.

Weather was cold but sunny, practically no wind. A really nice running weather. Close to the office there was still some traffic, so it was not so easy to have a conversation. But after the first couple hundred meters we reached a calmer grounds and the conversation was easy.

Another factor to keep in mind is pace. Don't go too fast or you will be just huffing and puffing. With a decent pace you can talk, but still get some exercise.

During the jogging we discussed some memorable events from the past couple of weeks and how's the cooperation with other team members and rest of the organization. But since these one-one-one's are my substitute for development discussion, I wanted to also discuss future. For most this is a difficult topic (it is the same for me too), but I think it's also really important. I demand Product Managers to share their Roadmaps, because it means that we have some plans for the future and we won't be just hitting a wall after the current Release is finished. That's why I'd like people to also have plans for their future. Many times these plans can be beneficial from company's point of view also.

Skipping to the end: we reached the office safely, went to shower and got back to our computers. As an experience I think jogging meeting is definitely worth trying. Of course there are certain prerequisites, but if everything is matches and you feel like it, go for it! I'm pretty sure we'll try this again.

Facing the Inevitable

Layoffs, bankruptcies and people losing their jobs seems to be almost like a trend nowadays. For the previous generations it was possible to graduate, get a job and finally retire with a gold watch after working in the same company for all of your career. If someone now has something similar in mind it almost feels naive.

Of course there are always exceptions. And probably there's big variation between different companies. But I think everyone needs to admit that things happen faster than before. Fortunes are made and lost in mere days, maybe even faster. Robots handle transactions at the stock market. One of the big dilemmas seems to be that there's a lot more data available than before, but it seems increasingly difficult to distill the important messages from the background noise. Do we have time and wisdom to understand what we measure?

I have adopted some rules of thumb from what I've read. I don't anymore think that people will spend all their lives in the same company. For my own company and team I try to create best possible circumstances for them to work. Create a system that they can feel connected with and achieve something. Exchange their free time to a hopefully competitive salary (I don't have much power over this) and to tasks that are intellectually challenging, foster their creativity and when ever possible, can be identified as things that move the company towards some greater goal.

Another thing to consider is situation when company needs to let some people go. It always feels bad. But sometimes it is inevitable. I think about Nokia as an example. If you manufacture normal, non-smart phones and people don't buy them anymore, the situation is tough. There's no easy way to increase the sales. Maybe you can find a new market where people still would prefer these 'old school' phones, but that doesn't change the fact that globally the demand for them has gone down. And will not go up anymore in the foreseeable future. And if your organization consist mainly of experts in this field, management doesn't have a multitude of options. You can wait until your bank account is empty, but in the end the result will by inevitable and ugly.

In the end the choices are to go out of business or face the facts and adapt. Things that worked in the past do not work anymore and you need to come up with something new. You will need to develop new competencies and be bold enough to let go of the past.

For some this is a cruel message. For some it can be a wake up call. My best advice for everyone could be to keep challenging yourself and to learn new things. If you learned to do something 10 years ago and continue to do the same year after year... one day your services may not be needed anymore.

Stay curious and find out new things. Experiment. Stay agile and adapt to the changing environment. Learn everyday!

Build Length

Continuous Integration is maybe one of my favorite engineering practices. With the green/red build indication you always know that your code compiles. Or doesn't compile. This information is vital.

After compilation the next step is running automated tests. Best practice approach would be to run unit tests. Unfortunately it's not always so easy to write tests for single modules. Then the tests tend to become somewhat integration tests. But since in my case we haven't made very strict rules about these, I'll simply adopt the vocabulary from How Google Tests Software and simply call them Small and Medium tests.

After these shorter tests are run, the following round will include Large tests. These are end-to-end tests that tests the functionality in many realistic use cases or even integration between different products.

Going a bit more into details, the execution times for these tests one year ago was around 10 hours for all Small and Medium tests. They were executed using physical machines and the system running them was TeamCity. Or actually the tests were ran for both x86 and x64 bit programs. number of tests was around 5k for each bitness.

The Large end-to-end tests took also 8-12 hours, but they had not been automated. Also the visibility to these tests was rather poor. They were executed by the system team, but the results weren't transparently available. The best way to know whether the tests were passing or not was to ask from some team member.

The radical change happened when build machines were virtualized. Instead of having a few high clock rate physical machines we ended up in having a set of servers which we filled with Hyper-V virtual machines. Number of processors was set to 8. We also made some modifications to run the tests in parallel (they were previously all just run consecutively). CI system was moved from TeamCity to Atlassian Bamboo.

Build times (or build + running tests) dropped  significantly. One thing we also learned was that virtualization platform makes a big difference. With all the same settings VMware was almost 20% faster than Hyper-V. Good thing is that both can be scripted nicely using PowerShell. We created VMware build machines with 16 logical processors.

During the same effort we managed to get also our Large tests to run in parallel in our continuous integration system. This wasn't as straight forward, because the framework used fixed file names and the test runs interfered each other. But in the end all the problems were resolved and we got the tests running automatically.

After the changes our build + Small & Medium tests now take 30-45 minutes depending a bit on bitness and other factors. Large tests are executed in about 45 minutes. So, for any change done to the codebase we now get the following phases:

1. Changes are checked into continuous integration and compiled
2. Small and Medium tests are executed.
3. Large tests are executed.

In parallel with step 2 we have other builds that produce installer for manual testing. Build time is much shorter, so the test execution still dictates the build length.

Before our latest efforts the steps 1-3 would have taken almost 22 hours including some manual steps. Now they take around 1,5 h for any changes, things are fully automated and the results are transparently available for anyone anytime. Since nothing is ever enough we aim to go even further, but I think the current results are already worth a small celebration!

Tower of Hats

A few days ago I started thinking about all the different things I do at work. I have a main job, but in addition I do bunch of other things too. (This post is not about any Agile practices or management. If you are mostly interested in those posts, please feel free to stop reading now. But if you are interested to read how tall my pile of hats is, please continue.)

My main job is Manager, Software Releases. All the software that goes out goes through me. At least in theory. With 10+ teams it's sometimes a challenge. :)

I'm the Chief Scrum Master, owner of the Scrum framework (and how it's implemented in the company) and somewhat coach of our Scrum Masters. Fortunately I have another person who has recently started to assist me in this.

I'm the owner of the Release Process. How we package the software and use version control. This also means that I arrange release activities like planning, demo & retrospectives for all our products.

I'm the owner of Software Development Process and company wide Definition of Done. I try to come up with common conventions that allow us to have top notch code while still having creative freedom.

I'm the head of Release Team. We keep the infra structure in place and make sure build machines and version control work. Also we maintain other tools like JIRA and Confluence. I also pitch into this work, but I admit my team mates are more capable for doing that. And I'm actually really, really proud of that.

In addition to the supervisory duties I'm the Product Owner of the team. So I also get to prioritize the customer needs and bugs in the Backlog. With so many internal and external customers to serve I think it's a real handful.

I'm the owner of the Software Product Creation value chain. This means all the activities between having a customer need and actually filling those needs. SW Development and Release Processes are part of this value chain, but it also includes Issue & Configuration Management Processes and testing & development practices.

I'm a member of the Technology Unit's management team. We discuss unit's different matters with other Product Owners and country managers. Mostly the bigger decisions concerning the unit are done here. Actually I'm also the secretary of these meetings.

I'm an Internal Auditor. This isn't really a big burden since we don't have internal audits but couple of times a year. But it's interesting and offers a chance to examine other parts of the organization than just R&D.

During last strategy update I worked as the internal strategy facilitator. Mostly this meant making sure all work groups progressed. In a way it was really really educational. I learned tons of new things about the company and opened my eyes more to the business and long term goals.

I'm the substitute for Technology Unit head & also for the Quality Manager. Usually these things don't require any actions from me because the actual people are mostly present.

I'm a project manager for a Tekes funded research project DD-SCALE. Mostly I interact with researchers from Helsinki Univeristy or attend steering group meetings with all participants.

I think I'm still the vice shop steward. Haven't done anything regarding this in ages.

I'm a member of the company's leisure team. We arrange different activities for the employees in Finland.

I'm rather interested in the social media. I actually created the twitter account for the company and have been rooting for more external communication. How can the customers know about the cool new stuff if we don't let them know about them?

There. If I counted correctly, it's about 10+ hats or so. At least my head shouldn't get cold. Sometimes it just gets a bit dizzy for all the different responsibilities. But maybe this explains why I like to simply call myself the Jack of All Trades

Things I do on my spare time are out of scope. And too numerous to fit here. ;)

Loops of Learning

I'm currently reading a book about Organizational Patterns of Agile Software Development. In addition to the interesting patterns I was especially attracted to the chapter about anthropological foundations. The book also opened my eyes to see that processes alone are not enough. I've written about this before, but my view was reinforced once more.

Continuous improvement with processes usually only deal with Single-Loop Learning. We inspect the results of our immediate reactions and then make corrections. Usually this is ok for a team that is building software in iterations. It's about following the rules. But when we go a bit higher, it maybe isn't enough anymore.

The second level, Double-Loop Learning was introduced to me by the Lean Startup book. In that method we still make small modifications as often as possible to learn fast, but we also can choose to either pivot or persevere. Do we want to keep on chasing the selected goal or should we select another target? Already this felt to me like something really awesome. Someone has even described the ideas of Lean Startup as possessing super powers. (I wouldn't maybe go that far, but it's a good book.) On this second level we create more insights about our actions. It can be also characterized as Systems Thinking.

But I wasn't aware, at least consciously, of the next level: Triple-Loop Learning until now. Jurgen Appelo writes about it in his blog post and Thorsten Gragert's wiki site offers a nice a summary including the below figure. The 3rd level deals with principles. In organizational context I'd associate it with company values and learning about learning.

Figure adopted from www.thorsten.org

By digging a bit more I found this article about Learning organizations. Interesting concept. They have the following five main features:

• systems thinking
• personal mastery
• mental models
• shared vision
• team learning

I think I need to give this a lot more thought. And try to lead my organization further on this path. Maybe I will coin a term #BeyondAgile. :)

And finally, as a somewhat sidestep, I'll share a some tips I have spotted from many successful organizations:

1. Dream big
2. Hire the best people (who share your dream)
3. Stay out of their way
4. Share the success