Build Length

Continuous Integration is maybe one of my favorite engineering practices. With the green/red build indication you always know that your code compiles. Or doesn't compile. This information is vital.

After compilation the next step is running automated tests. Best practice approach would be to run unit tests. Unfortunately it's not always so easy to write tests for single modules. Then the tests tend to become somewhat integration tests. But since in my case we haven't made very strict rules about these, I'll simply adopt the vocabulary from How Google Tests Software and simply call them Small and Medium tests.

After these shorter tests are run, the following round will include Large tests. These are end-to-end tests that tests the functionality in many realistic use cases or even integration between different products.

Going a bit more into details, the execution times for these tests one year ago was around 10 hours for all Small and Medium tests. They were executed using physical machines and the system running them was TeamCity. Or actually the tests were ran for both x86 and x64 bit programs. number of tests was around 5k for each bitness.

The Large end-to-end tests took also 8-12 hours, but they had not been automated. Also the visibility to these tests was rather poor. They were executed by the system team, but the results weren't transparently available. The best way to know whether the tests were passing or not was to ask from some team member.

The radical change happened when build machines were virtualized. Instead of having a few high clock rate physical machines we ended up in having a set of servers which we filled with Hyper-V virtual machines. Number of processors was set to 8. We also made some modifications to run the tests in parallel (they were previously all just run consecutively). CI system was moved from TeamCity to Atlassian Bamboo.

Build times (or build + running tests) dropped  significantly. One thing we also learned was that virtualization platform makes a big difference. With all the same settings VMware was almost 20% faster than Hyper-V. Good thing is that both can be scripted nicely using PowerShell. We created VMware build machines with 16 logical processors.

During the same effort we managed to get also our Large tests to run in parallel in our continuous integration system. This wasn't as straight forward, because the framework used fixed file names and the test runs interfered each other. But in the end all the problems were resolved and we got the tests running automatically.

After the changes our build + Small & Medium tests now take 30-45 minutes depending a bit on bitness and other factors. Large tests are executed in about 45 minutes. So, for any change done to the codebase we now get the following phases:

1. Changes are checked into continuous integration and compiled
2. Small and Medium tests are executed.
3. Large tests are executed.

In parallel with step 2 we have other builds that produce installer for manual testing. Build time is much shorter, so the test execution still dictates the build length.

Before our latest efforts the steps 1-3 would have taken almost 22 hours including some manual steps. Now they take around 1,5 h for any changes, things are fully automated and the results are transparently available for anyone anytime. Since nothing is ever enough we aim to go even further, but I think the current results are already worth a small celebration!

Tower of Hats

A few days ago I started thinking about all the different things I do at work. I have a main job, but in addition I do bunch of other things too. (This post is not about any Agile practices or management. If you are mostly interested in those posts, please feel free to stop reading now. But if you are interested to read how tall my pile of hats is, please continue.)

My main job is Manager, Software Releases. All the software that goes out goes through me. At least in theory. With 10+ teams it's sometimes a challenge. :)

I'm the Chief Scrum Master, owner of the Scrum framework (and how it's implemented in the company) and somewhat coach of our Scrum Masters. Fortunately I have another person who has recently started to assist me in this.

I'm the owner of the Release Process. How we package the software and use version control. This also means that I arrange release activities like planning, demo & retrospectives for all our products.

I'm the owner of Software Development Process and company wide Definition of Done. I try to come up with common conventions that allow us to have top notch code while still having creative freedom.

I'm the head of Release Team. We keep the infra structure in place and make sure build machines and version control work. Also we maintain other tools like JIRA and Confluence. I also pitch into this work, but I admit my team mates are more capable for doing that. And I'm actually really, really proud of that.

In addition to the supervisory duties I'm the Product Owner of the team. So I also get to prioritize the customer needs and bugs in the Backlog. With so many internal and external customers to serve I think it's a real handful.

I'm the owner of the Software Product Creation value chain. This means all the activities between having a customer need and actually filling those needs. SW Development and Release Processes are part of this value chain, but it also includes Issue & Configuration Management Processes and testing & development practices.

I'm a member of the Technology Unit's management team. We discuss unit's different matters with other Product Owners and country managers. Mostly the bigger decisions concerning the unit are done here. Actually I'm also the secretary of these meetings.

I'm an Internal Auditor. This isn't really a big burden since we don't have internal audits but couple of times a year. But it's interesting and offers a chance to examine other parts of the organization than just R&D.

During last strategy update I worked as the internal strategy facilitator. Mostly this meant making sure all work groups progressed. In a way it was really really educational. I learned tons of new things about the company and opened my eyes more to the business and long term goals.

I'm the substitute for Technology Unit head & also for the Quality Manager. Usually these things don't require any actions from me because the actual people are mostly present.

I'm a project manager for a Tekes funded research project DD-SCALE. Mostly I interact with researchers from Helsinki Univeristy or attend steering group meetings with all participants.

I think I'm still the vice shop steward. Haven't done anything regarding this in ages.

I'm a member of the company's leisure team. We arrange different activities for the employees in Finland.

I'm rather interested in the social media. I actually created the twitter account for the company and have been rooting for more external communication. How can the customers know about the cool new stuff if we don't let them know about them?

There. If I counted correctly, it's about 10+ hats or so. At least my head shouldn't get cold. Sometimes it just gets a bit dizzy for all the different responsibilities. But maybe this explains why I like to simply call myself the Jack of All Trades

Things I do on my spare time are out of scope. And too numerous to fit here. ;)

Loops of Learning

I'm currently reading a book about Organizational Patterns of Agile Software Development. In addition to the interesting patterns I was especially attracted to the chapter about anthropological foundations. The book also opened my eyes to see that processes alone are not enough. I've written about this before, but my view was reinforced once more.

Continuous improvement with processes usually only deal with Single-Loop Learning. We inspect the results of our immediate reactions and then make corrections. Usually this is ok for a team that is building software in iterations. It's about following the rules. But when we go a bit higher, it maybe isn't enough anymore.

The second level, Double-Loop Learning was introduced to me by the Lean Startup book. In that method we still make small modifications as often as possible to learn fast, but we also can choose to either pivot or persevere. Do we want to keep on chasing the selected goal or should we select another target? Already this felt to me like something really awesome. Someone has even described the ideas of Lean Startup as possessing super powers. (I wouldn't maybe go that far, but it's a good book.) On this second level we create more insights about our actions. It can be also characterized as Systems Thinking.

But I wasn't aware, at least consciously, of the next level: Triple-Loop Learning until now. Jurgen Appelo writes about it in his blog post and Thorsten Gragert's wiki site offers a nice a summary including the below figure. The 3rd level deals with principles. In organizational context I'd associate it with company values and learning about learning.

Figure adopted from www.thorsten.org

By digging a bit more I found this article about Learning organizations. Interesting concept. They have the following five main features:

• systems thinking
• personal mastery
• mental models
• shared vision
• team learning

I think I need to give this a lot more thought. And try to lead my organization further on this path. Maybe I will coin a term #BeyondAgile. :)

And finally, as a somewhat sidestep, I'll share a some tips I have spotted from many successful organizations:

1. Dream big
2. Hire the best people (who share your dream)
3. Stay out of their way
4. Share the success

Scales and Emergent Properties

It is fascinating how things look different depending on the distance. From space Earth looks like a blue ball with some green stuff here and there. By zooming closer one can see the continents and mountains and some of the biggest man-made structures like highways. Going closer still and taking a helicopter view the individual houses are yet indistinguishable, but cities have forms.

Humans can't usually see the forest for the trees. Trees are closer to our scale and we usually look at them from the ground up, not from top down. Also we and all the organic material around us consists of small cells. Many of them have a specific purposes (like muscle or brain cells). They are visible with a microscope.

If you could magnify even more, you would see that even the cells aren't the ultimate building blocks. They could be further divided into atoms which could be further divided into quarks (and maybe even further, but let's stop here.)

Many of these levels have their own forces and interactions. Quarks can feel the strong interaction (or nuclear force). In their world, gravity or electromagnetism isn't very interesting. On the other hand, interactions and events on the cosmic scale take so long and distances are so big that compared to them the human lifespan is only a drop in an ocean.

The same fascinating scale related properties exist also in the corporate world. Individual people, teams, units and companies work on different scales. After saying this I do admit that some exceptional individuals might match the output of a whole team or even a small company. But I'd claim that there are such efforts that simply aren't doable without the appropriate size. And yes, on the other hand bigger scale usually makes things more stiff and slower.

I think a team has a lot better chances to really finish a software feature. And I mean the whole deal: thinking about the usability, implementation, documentation, testing, optimizing, refactoring and packaging everything into an appealing form. And if we go a bit further and think about productization of an idea, I think a company is better off than a team or an individual. Sales and marketing are essential activities. But such capabilities aren't often found in the standard developer skill set. For a company to be successful, both skills are needed. But individual people can be highly successful in their roles with only one of these skills.

No-one can do everything. By working together we can make great things happen. With the joint effort of brilliant minds we have been able to reach the space and descend into the deepest ocean depths. If we could work together as a planet who knows where we would end up... But currently I'm happy to follow the road ahead with my company. :)

Release Planning in One Day

In this blog I have shared experiences of our Release Planning Days. Of course the first one was most memorable since back then everything was new and exciting. Second attempt proved that it wasn't pure luck, but that the concept was actually working. I documented also the third one, but after that I thought that it maybe isn't that exciting anymore.

One unfortunate thing to mention is that the day was a public holiday in one of our offices. But quite a few of our colleagues from that site were willing to come to work on that day and get another day free in exchange. I call that going the extra mile.

But maybe I can tell about the most recent attempt, since it was again a bit different from the previous times. For our latest Release Planning I had given some of my colleagues a few chores to fill in advance. Here's the list:

• Portfolio Backlog is in presentable state
• Product Managers have discussed their the Roadmaps with teams working directly with them
• Product Managers have made an initial Launch Plan for their products
• Product Owners have crafted Draft Release Plans for their teams

Out of these, the first task went to our Chief Portfolio Officer who is like the Grand Vizier of the Portfolio level. Second and third tasks went to Product Managers (Program level) and the fourth one to the Product Owners (Team level). 

Launch Plan is a term that isn't originally from SAFe (as far as I know). It communicates the intended usage of the Release if and when it would be successful. Previously we had identified a waste of extra waiting time when the Release was ready, but we had not decided what to do with it. The accompanied business needs are in most cases clear already when the work on the release starts. That's why it is practical to craft also a plan about the usage. And if something is left our the Release scope, the plan can be adjusted.

So, in essence we skipped the first Team Breakout. Also the organization of the presentations was modified. In our past Release Plannings all Product Managers have presented their Roadmaps in a row. But since we had the Draft Release Plans already available at the beginning of the event, I wanted to form logical entities of the Roadmaps and associated Release Plans. This also helped in focusing on one subject at a time.

The morning part of the day was heavy. That was clear already beforehand. But we didn't come up with any better alternatives. At least grouping the presentations in the new way was seen as an improvement.

All in all things went pretty much as expected. During the Team Breakout some plans were modified but most of them were almost the same as during the draft phase. The new, more compact way of conducting the day seemed to work well and we will probably do it like this also next time.

Answers to Open Questions

This time I'm writing a little differently structured blog post. Pierre Godts asked me the following questions at the Scaled Agile Framework group at LinkedIn. The question is related to the SAFe Case Study I wrote some time ago. So let's dive a bit deeper into some topics that were left our of the study.

Can you tell us more about the change at employee level? 

The change touched mostly people in the development. Previously all developers (coders and testers) had been in one big group and everyone shared the same supervisor. Work was organized around projects that were resourced from this pool of developers on a monthly basis. Projects were steered by people working in the business units.

After the change people were assigned to (Scrum) teams. Each team was appointed a Product Owner who was a member of a business unit. POs reported to the heads of business units (Chief Product Owners or Business Owners). Teams started to follow scrum way of working. During the same time our internal processes were also identified and written down. Release cycle was set to three months although we knew that we would probably face problems in the beginning.

Each team elected their own Scrum Master who then started to facilitate the events and see that we followed the scrum ceremonies. They were all certified (as were the POs). Scrum Master Community of Practice was also established and Chief Scrum Master started to facilitate it.

At the beginning Scrum Masters participated in a weekly Scrum of Scrums meeting. This was later replaced by a meeting that was between any member of each team. Scrum Masters started to have regular get-togethers where they discussed process related affairs and possible impediments that needed to be raised to company level.

What was their commitment to change? Were employees skeptical or not willing to change? 

People are all different. Some were more skeptical and some were more like early adopters. I belong to these guys who usually get excited about new things easily. But generally I'd say people were rather committed to the change. The previous state of the company was not really optimal either so I think there was a really fertile ground for some change.

And as depicted in many change management related books the early and late majority just needed a bit more time and discussion. Maybe with some more coaching the change could have been faster. In our case teams were more or less left to figure out the steps themselves. But maybe we just did it with some luck.

Did some employees got fired? 

No. No employees got fired during the reorganization. But some didn't like the new way of working and decided to seek their destiny somewhere else. Some started their own company, some simply changed employer. I claim that quite many of these people had been thinking about the change already for a bit longer while. The reorganization was just the final nail into the coffin.

But one really positive thing is that some people have returned. They have seen the greener grass but understood that maybe it's due to more fertilization...

Were new employees hired? 

Yes. The company has been growing and new teams have been established during the past years.

Was there an increase in number of employees? 

Yes. Same as previous question.

What about stress level, increased or decreased? 

This is tough one. Depends. No-one probably misses the long death marches when we tried to get the release together after a year of development. Or the feeling that if I don't get this feature into the product the customers will need to wait for it for another year. (This would inevitably increase the scope creep and result in not so well tested additions at the really late part of the release cycle.)

But then the other side of the coin is that previously some developers had a really big areas of responsibility. This would mean that you could potentially save the whole company by fixing some issues and get credit from everyone. When you are working in a team, it's usually the team that gets the credit. But I see this again as one inevitable thing that happens when a company grows over certain limit.

What about the budgets/costs, controlled? 

Honestly I don't know the answer to this one. Probably the budget was controlled, but it was done in so smooth way that it was never evident. More I would say people didn't take the full benefit of their empowerment. When you have been just a resource and someone else has been managing you or your project for long enough, you don't really understand what it means when your boundaries are removed. It's a slow process to get people to see that "Hey, can I do this? Ok, wow, I can!" And it feels great. As an employee it's a bit scary. But as a supervisor I think it's great to empower others!


Did the use of Jira cause problems/difficulties?

Not really. With our distributed setup JIRA was really an essential tool. I like physical boards and post-its but with multiple sites it's a luxury you can't really afford. And when you accompany JIRA with other Atlassian tools you can get real transparency to the development. You can see whole chain from idea to source code to testing and finally to the product. And with documentation included.

It Yourself Framework Do

As the saying goes, one picture tells more than 1000 words. Maybe that's why there are these comprehensive frameworks that offer nice pictures that you can click and get even more information. There's one in SAFe. And another one in LeSS. And of course DAD has one too. After checking all these others I was actually a little disappointed that Pragmatic Marketing framework does NOT open up the same way. (Improvement suggestion?)

Since many of these frameworks are copyrighted and no modifications are allowed I'm not going to propose anything like that. But, if you have internal processes or any other information that is dispersed in multiple locations and you would like to visualize them in a similar way with these frameworks I'm going to teach you how to make a poor/lazy man's version of the same. You will need the following tools:

• PowerPoint
• Some picture 
• Documents located at some internal/external URLs

Because today is #StarwarsDay (#MayThe4thBeWithYou), let's use something related as an example.

First, create a new presentation and add your favorite to-be-framework-background picture to the slide.

Then add the first hyperlink location. If you want to use really fancy shapes, be my guest, but I'd recommend simple ovals and squares.

Draw the shape on top of your background image. Then modify both Shape Fill and Shape Outline to be invisible (No Fill, No Outline).

After this you simply add an Action to the shape that opens the specific URL. I recommend also using the highlighting to more easily spot the actionable area.

 

Because I had nothing special in my mind I just used Google to find out what can be found with 'Yoda's Eye'. Turns out there is a metal band with this name. After inserting this link and starting my ppt-presentation the result is the following:

The same step for all your relevant links you repeat and a really actionable internal framework get you will! May the force be with you!