Build Length

Continuous Integration is maybe one of my favorite engineering practices. With the green/red build indication you always know that your code compiles. Or doesn't compile. This information is vital.

After compilation the next step is running automated tests. Best practice approach would be to run unit tests. Unfortunately it's not always so easy to write tests for single modules. Then the tests tend to become somewhat integration tests. But since in my case we haven't made very strict rules about these, I'll simply adopt the vocabulary from How Google Tests Software and simply call them Small and Medium tests.

After these shorter tests are run, the following round will include Large tests. These are end-to-end tests that tests the functionality in many realistic use cases or even integration between different products.

Going a bit more into details, the execution times for these tests one year ago was around 10 hours for all Small and Medium tests. They were executed using physical machines and the system running them was TeamCity. Or actually the tests were ran for both x86 and x64 bit programs. number of tests was around 5k for each bitness.

The Large end-to-end tests took also 8-12 hours, but they had not been automated. Also the visibility to these tests was rather poor. They were executed by the system team, but the results weren't transparently available. The best way to know whether the tests were passing or not was to ask from some team member.

The radical change happened when build machines were virtualized. Instead of having a few high clock rate physical machines we ended up in having a set of servers which we filled with Hyper-V virtual machines. Number of processors was set to 8. We also made some modifications to run the tests in parallel (they were previously all just run consecutively). CI system was moved from TeamCity to Atlassian Bamboo.

Build times (or build + running tests) dropped  significantly. One thing we also learned was that virtualization platform makes a big difference. With all the same settings VMware was almost 20% faster than Hyper-V. Good thing is that both can be scripted nicely using PowerShell. We created VMware build machines with 16 logical processors.

During the same effort we managed to get also our Large tests to run in parallel in our continuous integration system. This wasn't as straight forward, because the framework used fixed file names and the test runs interfered each other. But in the end all the problems were resolved and we got the tests running automatically.

After the changes our build + Small & Medium tests now take 30-45 minutes depending a bit on bitness and other factors. Large tests are executed in about 45 minutes. So, for any change done to the codebase we now get the following phases:

1. Changes are checked into continuous integration and compiled
2. Small and Medium tests are executed.
3. Large tests are executed.

In parallel with step 2 we have other builds that produce installer for manual testing. Build time is much shorter, so the test execution still dictates the build length.

Before our latest efforts the steps 1-3 would have taken almost 22 hours including some manual steps. Now they take around 1,5 h for any changes, things are fully automated and the results are transparently available for anyone anytime. Since nothing is ever enough we aim to go even further, but I think the current results are already worth a small celebration!

Building a New Product

As my opinion about perfect team structure has developed during the past years, so has my understanding of how awfully many different things need to be considered when building a new (desktop) software product.

As a coder I used to think that a random idea is a good starting point for a new feature. I still believe that making a crude and fast proto is a good idea, but there's maybe even better way. Before writing any code, make a sketch. Before implementing any functionality, test the idea with a real customer if possible. Then make variations to the design and iterate again. Do this for a while and you have less code to maintain and better understanding of what you maybe should be doing (Lean Startup and Lean UX).

Instead of just copying files between different computers, you might want to set up a version control system. There are many providers, but I can easily suggest Git. It's free, relatively easy to learn and so widely used that you can get help and skilled Git users easily.

"Well written code is its best documentation". I partially sign this statement but not totally. Useless comments that get out of date are just useless. But there might be general design ideas like what kind of inputs and outputs are expected and how different parts of the product are supposed to communicate together. Usually these higher level characterizations don't change much even though classes and modules can have much shorter life cycles. So some amount of (design) documentation is nice.

I like to promote test-driven development, because then you get a nice set of unit tests and presumably your life will be much easier later. It's like an investment into the future. Of course before you can write tests you need to implement a test framework. Fortunately for many modern languages these are often easily available. But if you are dealing with more exotic playground, you might need to invest a bit more time and effort. Never the less, having a unit test framework is an investment you seriously want to make.

It doesn't harm if testing and test automation is given a thought already at the design phase. You might want to have a tester involved.

If you are not intending to make your product open source (which is of course ok too) you should maybe build some kind of licensing system. There are many possible business models and maybe it also depends on if the users are expected to have internet access or not. But it's also an aspect to keep in mind.

Last but not least, after developing the product for a while you might remember that there should also be a way for the users to install the software. And it might have some prerequisites and dependencies that should be fixed before the happy user is able to start your product. You need an installer (WiX, NSIS, InstallShield). And if you haven't done that yet at this stage, you probably also want to set up a Continuous Integration system.

In addition to these, there are many more useful practices (coding and documentation conventions, manuals, 3rd party libraries, etc.), but I think I'm done here. The main point of this post was that there are really many things to consider when kicking up a new product. And even if you implement these it doesn't really guarantee anything yet. But at least you are not failing because you missed these things. :)

Do the Right Things Right and Fast

It's fascinating how sometimes some relatively simple picture can contain so much relevant information. I have been lately involved in strategy renewal and while doing that I saw a picture that I fell in love with.

My job is very much concentrated around improving software development processes. In essence it's about doing the things right. But many times I have thought that it doesn't really matter if you do things right if they are not the right things. You might write the most beautiful and simple code ever, but it makes no difference if no one cares about it. I feel this is often a problem with some open source projects. Okay, there might be open source with crappy code quality too, but usually the quality is on a really nice level.

Another case are commercial projects with tight deadlines. In those cases it's easily right things (someone, usually customer, is really interested about the results), but just fast. When coders are stretched, it is the quality that suffers.

By concentrating on the software production methods (Continuous Integration, Continuous Delivery) and good engineering practices (craftsmanship), one can only reach adequate results. One dimension is still missing.

Maybe the best non-perfect position to be is when you have a good portfolio (you are doing the right things) and you concentrate on your code quality (doing the things right). At this point you can still miss valuable market windows, but it can still work. Then you can start investing in the production methods and speed things up until you hit the sweet spot in the intersection of all the three dimensions.

I thing this later triplet of circles depicts the situation at practical level. When your Product Management is working properly, your Developers are applying good engineering practices and your development infrastructure is in order, you can expect awesome results. I'm not saying it's easy to achieve but definitely worth a try.

DIY Build Light Indicator

This post will differ from my typical posts. It's not about processes or methods. But actually it's related to the practice of Continuous Integration. That's one of the key engineering practices most Agile teams use. Originally proposed by Grady Booch, but later popularized as one of the eXtreme Programming practices. For many Continuous Integration is simply "that build system".

There are many free and commercial tools available for CI. I have actually played around with CruiseControl (not much and I've never actually configured any builds in CC. It was more or less fading away when I joined my company), TeamCity (this one I like a lot), Jenkins and lately Bamboo.

The problem with all these CI tools, and all other online tools, is that you need to navigate to a specific place to find out if the build is passing or not. Similar thing for a Scrum wall or Burndown chart if you use for example JIRA. For us physical humans the physical things just work well...

The build light is a remedy for this. It converts the status of the build in our CI tool to a physical form: colored light. If the light is green, the build is passing. If the light is red, someone needs to fix the build ASAP! This kind of information radiator can easily spark action and foster self-organization.

Making yourself a build light isn't that difficult nowadays. In our case the most difficult thing was to get a Phillips hue Starter kit. With that you get a light that you can control with your computer. Then you will need a CI tool. In our case we connected our build light to a Bamboo build.

Then maybe the most difficult part: you need the glue code between your CI and your lamp. But today that's not too difficult either. Node-RED has made the task of connecting our physical devices to internet lot easier. (I'm not sure if it was difficult before. I'm just assuming it was.) Instructions for connecting your hue to Node red can be found from this blog post. (Actually before you get to use Node-RED, you will need node.js installed.)

But skipping ahead a few steps you get this: voila! Now we can diminish the "is the build passing?" questions. Simply check the lighting. If it's pleasant green you have nothing to worry. :)

Our buildlight. Who broke the build!?

(Release) Version Control

During my holiday I noticed that Scaled Agile Framework had been updated to version 3.0. I did watch the short update video by Lean Samurai (which was nice and compact), but I haven't really had time to get familiar with most of the contents yet. Some positive first observations include dropping Hardening out of the HIP-sprint (nowadays only Innovation-Planning) and making an attempt to include also strategy into the big picture. Although I like IP more than HIP, I don't really like dropping the shippability from the PSI. I'm glad that separation between development and releasing is still there. Maybe I will learn to like Program Increments.

But the topic I was about to write today is Continuous Integration. It has now been lifted to the big picture to get the attention I think it really deserves. Scrum doesn't offer engineering practices (XP does), so companies should amend their processes with some additional common agreements. Let me now write down some clichés that have some valid points behind them:

• You can't scale crap.
• You can't go fast if you carry a lot of extra.
• Integrate often.

Well, one could consider these also as plain facts. But the first two are more or less related to good coding practices and craftsmanship and staying Lean. Also keeping the technical debt as low as possible helps going faster. The last bullet is related to version control and software production.

We are applying the stable mainline pattern. To get a more thorough explanation on this please check here or here (again I find good stuff and it has Mr. Kniberg's name written on it. That guy is a wizard!) We use a rather well integrated set of Atlassian tools. Atlassian Stash as the basis of our repository management accompanied with Atlassian Bamboo crunching the builds. And they both communicate nicely with JIRA. (While looking for some nice pictures, I bumped into this video about making a demo integration. Maybe worth watching.)

When the code changes are checked in, the builds start automatically. They include running the automated tests (unit and integration), followed by more wider range system level tests if everything went well. Good changes are blessed with a nice green color, but any breaking changes result in an ugly red color.

In Bamboo you can configure the plans to make builds for all new branches automatically. This makes the creation of release branches handy (compared to some other CI frameworks where you need to configure a new build for each new branch.)

Finally, the big dilemma when we make releases. We branch the release from the trunk. This is communicated well in advance to all the teams and developers. But after we make the branching, we don't allow any more changes to the code. Stash offers a handy way to do it from the branch permissions. To be more precise, we do allow changes, but they need to be made via pull-requests and approved by the System Team.

My dilemma here is that I'd like to believe in people. I believe most of the restrictions usually hinder development. But then again, I believe many people are aware of the term feature creep. It's easier to fight it if you don't have a way to make changes. And the obligatory pull-requests have a built-in code review which usually helps improve the general code quality.