How to Set Up a System Testing Team

In my previous post I touched the topic of legacy modification. It is a difficult stunt to pull. Or well, in a perfect world with a well maintained codebase and outstanding automated test coverage it could be easy, but in reality I think that's rare.

One of the things that makes modifying legacy software especially difficult is the regression. When the system is strongly coupled, almost every little change can produce unforeseen consequences. (Yes, the Butterfly effect!) To counter-attack the diabolic regression effects your best bet is to invest in a test automation harness. But until your test harness is truly solid, you need to make do with manual testing.

In our setup we have two distinct periods: the Development period and the Stabilization period. During the Development period our scrum teams work normally in sprints. They concentrate on creating the new added value that has been agreed on in the Release Planning. Adding new code into the mainline is guarded with automated tests (unit tests + system level tests) and only allowed through pull-requests. Unfortunately due to the limited coverage, this is not enough to guarantee that things stay in good shape.

Development

Development

Stabilization

During the Stabilization period we stop adding new features and concentrate on testing the existing functionality. In practice this means testing manually those parts that are not covered by the automated tests. (This might be called classic hardening anti-pattern and frowned upon, but in this setup it's a must. New development meets Definition of Done, but it's the regression that gives us headache.)

Now this setup sounds a bit dumb. Why do we let the quality deteriorate and problems creep into the mainline and wait to be uncovered during stabilization? Wouldn't it be wiser to detect the problems early on?

Well, that's why we came up with System Testing virtual team. At least now in the beginning it consists of each teams' tester and scrum master. Reasoning behind this resourcing is that testers should naturally be interested in their product's overall quality. And unreleasable software is a really big impediment, thus the scrum masters. The concept is actually quite close to Scrum of Scrums. One main benefit is also that the teams will become aware of what the others have been doing.

The virtual team will meet biweekly for a full day of testing. Each day will be like a mini sprint: beginning with a planning session and ending with a review and retrospective. There's still plenty of forming and storming to do, but I have really high hopes for the activity. Already during the first session they were able to catch tens of bugs that would have most probably otherwise stayed under the radar (at least) until the Stabilization period!