How to Set Up a System Testing Team

In my previous post I touched the topic of legacy modification. It is a difficult stunt to pull. Or well, in a perfect world with a well maintained codebase and outstanding automated test coverage it could be easy, but in reality I think that's rare.

One of the things that makes modifying legacy software especially difficult is the regression. When the system is strongly coupled, almost every little change can produce unforeseen consequences. (Yes, the Butterfly effect!) To counter-attack the diabolic regression effects your best bet is to invest in a test automation harness. But until your test harness is truly solid, you need to make do with manual testing.

In our setup we have two distinct periods: the Development period and the Stabilization period. During the Development period our scrum teams work normally in sprints. They concentrate on creating the new added value that has been agreed on in the Release Planning. Adding new code into the mainline is guarded with automated tests (unit tests + system level tests) and only allowed through pull-requests. Unfortunately due to the limited coverage, this is not enough to guarantee that things stay in good shape.

Development

Development

Stabilization

During the Stabilization period we stop adding new features and concentrate on testing the existing functionality. In practice this means testing manually those parts that are not covered by the automated tests. (This might be called classic hardening anti-pattern and frowned upon, but in this setup it's a must. New development meets Definition of Done, but it's the regression that gives us headache.)

Now this setup sounds a bit dumb. Why do we let the quality deteriorate and problems creep into the mainline and wait to be uncovered during stabilization? Wouldn't it be wiser to detect the problems early on?

Well, that's why we came up with System Testing virtual team. At least now in the beginning it consists of each teams' tester and scrum master. Reasoning behind this resourcing is that testers should naturally be interested in their product's overall quality. And unreleasable software is a really big impediment, thus the scrum masters. The concept is actually quite close to Scrum of Scrums. One main benefit is also that the teams will become aware of what the others have been doing.

The virtual team will meet biweekly for a full day of testing. Each day will be like a mini sprint: beginning with a planning session and ending with a review and retrospective. There's still plenty of forming and storming to do, but I have really high hopes for the activity. Already during the first session they were able to catch tens of bugs that would have most probably otherwise stayed under the radar (at least) until the Stabilization period!

Paved with Good Intentions

How Local Optimization May Result in Failure 

Sometimes when you remove a rusty bolt from a system of nuts and bolts and clean it up, it may not fit the original context anymore. In the course of time the rust has become part of the system and should be also taken into account.

In a complex software legacy system this phenomenon may manifest when a certain subsystem is refactored or rewritten. Technically the new part may be superior and fulfill the original specifications, but the changes may still result in nasty regression issues. During the past years the layer of application logic has been built on top of the faulty behavior (the rust).

Same applies in organizational context. High-performance team may outrun the system around it. It may produce results faster than the other parts can consume. This can create queues and waste. Results can be even worse when the sub optimization happens on the unit level. For example when R&D is able to produce results faster than can be delivered or specified.

Usually increasing performance is desirable, but some gotchas should be avoided. If you  want to remedy the possible problems, I’d suggest a system view. Try to improve the overall flow of value through the system. Tweak the parts, but don’t fall for the local optimization. Aim to decrease the lead time and measure things from the customers’ perspective.