Scaling Software Development

Software development is a team sport. It is true that even a single person can make huge contributions (like Linus Torvalds on Linux), but in most cases great results are produced by great teams. In this post I will concentrate on agile software development in a setup where own products are being made. I will try to tell how I would extend the team and what kind of tools or practices could be used.

Staffing the first team

 

Product Owner

Everything starts with an idea. A good rule of thumb is that there's a problem significant enough that it is worth solving. Before coding even a single line of code, make a hypothesis of the possible solution. Then validate it as fast as possible. For example Lean Startup practices might serve you in doing this. Having a Product Owner (PO) might be a good idea. PO can start collecting potential product feature ideas and requirements into a Product Backlog.

Architect

When you are pretty sure that you have an idea worth implementing and you start creating a team, maybe the first person to recruit is the architect. Someone who will have an overall technical vision about structure of the solution.

Coders

Next you probably want to add a few more coders to speed up the development. In the early phase of the product lifecycle it's tempting to divide the responsibilities so that there's a single person responsible for a part, but I would advise against doing so. It will limit the shared code ownership and most probably create bottlenecks. All parts of codebase should be familiar to more than one person.

Design and testing

Depending a bit on the case, next I would get a designer or a tester. If the solution needs to be operated by a user, I'd go with designer. If most of the magic happens in the backend and no-one cares how butt-ugly the product is, then first concentrate on making sure things work. Don't get a tester, but a person who is familiar with testing and can create automated tests. But testers think differently than coders, so better get one onboard.

Now you probably have quite standard software development team. Either get someone who has prior knowledge about agile software development to act as scrum/kanban master or select a person from the team that is most interested to learn the topic. With the help of this person the team will start an exodus to become a self-managing, self-organising cross-functional team that turns Product Backlog items into working software.

Scaling up

 

Splitting

Natural way to scale is to increase the team size by recruiting more and more coders and maybe domain experts. Once the team is so big that regular meetings like daily scrum are no longer efficient, it's time to start thinking about how to split the team.

When you split the team, think carefully how the new teams divide the responsibilities. Let the tech people decide what suits them best. When you have the second team, make sure that also contains all needed competences, including test automation.

Centralized competences

Some roles might not be needed full time in the development teams. It might make sense to centralize these to a different team that can support multiple dev teams. Architecture and design can be such skills that they can be in a common team.

When the work is divided between multiple teams it is also a sign that your operations need to raise to a new level of professionalism. It is a good idea to recruit someone to take ownership for the documentation. Security is also more important and maybe worth having someone concentrating on it. And since there are multiple teams (possibly) working on the same codebase, someone should spend a bit more time on setting up proper integration and deployment practice. Also it is worth thinking about how the teams coordinate and synchronize their work. Some scaling framework could be beneficial. For me Scaled Agile Framework is the most familiar. It also provides guidance on how to plan and execute things when you have multiple teams and also implement continuous improvement.

Get going

When you have recruited people to fill more than one development team and staffed another team has some centralized capabilities, it is time to plan how you shift the gear. This part requires a bit more effort on planning and discussion. I find it practical to have a person write down instructions that offer not too much, but enough details about how the new setup works. Then it is time to just agree on starting day and get going. Execute the initial plan, stop to reflect how did it go, adjust and repeat. That's how iterative and incremental practices work.

Tools

There are some tools that have become almost industry standard for supporting agile software development. Some tools have a lot of different features integrated to them and can handle a big part of the development pipeline. But I will give here an example of pretty traditional setup.

Backlog handling

For handling the Product Backlog, a pretty easy choice is Atlassian Jira. You can create and prioritise backlog items, track bugs and follow progress. If you are using Scrum, Jira offers possibilities for having burn down charts for your sprints. You can also easily visualise the workflow with kanban board and customize the columns to reflect your development process steps (like coding, code review, testing, etc.)

Version Control

The only version control tool that I would recommend is Git. There are multiple commercial tools that you can use that offer different additional benefits. Here are a few: GitHub, BitBucket, GitLab. Access rights control to the source code is probably the main thing, but maybe next in line is the possibility to use pull requests. Instead of pushing things straight to master you should consider a workflow that fosters shared code ownership and transparency. Code should be reviewed by someone else than the writer before merging.

Continuous Integration

Some Continuous Integration (CI) system will help you to automate your process. If the code needs to be compiled, the system will do it. In addition it can start the execution of automated tests, usually starting with unit tests. Some examples of CI systems are Jenkins, Drone, Bamboo, CircleCI, TeamCity and Travis.

Static Code Analysis

For static assessment of your codebase quality I would recommend adding a tool like SonarQube to your pipeline. It can help you measure how readable your code is and how good your test automation harness is. It will cover also security vulnerabilities.

Deployment 

Then you probably need some place to deploy your solution. Either you invest in hardware or you use cloud services. The most common public cloud platforms are AWS, Azure and Google Cloud. If you are using containers, you can orchestrate them with Kubernetes. If you can spend a bit more bucks, you can use an enterprise solution like OpenShift.

Documentation

Quite a standard way to build and maintain living documentation is through Confluence. Pages have version control and editing is very similar as in Wikipedia. They can be exported in multiple formats, such as pdf, word or html.

Possible pitfalls

 

Recruitment

If creating awesome applications and products that users love would be easy, everyone would do it. But it's not so simple. For starters, there's a shortage of skilled workforce. This means that people who know the craft also know their value. And since the whole world is getting digital, there are lots of companies that are willing and able to fight for the talent and offer high salaries and different perks. So, even if you can recruit a team, you might lose members because some other company 'buys them out'.

Culture

Another thing to understand is that the results that an organisation is able to produce are a lot more dependent on how well the people work together than how good the individuals are. It doesn't mean that you can achieve great results with mediocre people. But it does mean that even with highly skilled persons you can fail miserably if they don't get along. Culture makes a huge difference. If people feel safe to show their vulnerability and admit mistakes, everyone will benefit. With retrospectives and conscious continuous improvement efforts teams and teams of teams can raise to a whole new level of performance.

Edit 1: Added some sub-headers for easier navigation of topics.

Story about Software Lifecycle

This is a tale of a software lifecycle. It's fiction, but into what extent - I won't tell. It has been inspired by real events.

The Beginning 

About seven years ago a big consumer company wanted to renew their customer web portal. They didn't have own software development department, so they hired a team of external consultants for the task. Consultants came from two different companies. One was a smaller company with senior specialists. They did the architectural design. The other company was bigger. Already from the beginning the plan was that the bigger company would maintain the portal. Their developers were less senior, but the plan was that they would learn during the building phase.

The portal was a big success. It even won prices. When the intensive building period was over, the smaller consultancy company went to do other things and the other company began to maintain the portal. From now on I'll call them supplier (or team A).

For quite a few years everything went well. there was one person on the customer's side who was in close contact with the supplier. Supplier had a dedicated team and an architect/scrum master. New features were being built and things worked smoothly.

Maturity

Time passed. The customer relationship was so intense that at one point the architect needed to blow the whistle and take some time off. The person was replaced by another architect. The developers who were involved in the early phase were also in the early phase of their careers. And it is quite understandable that after a few years they wanted to do something else. Some changed roles, some went to work in other companies. If you are familiar with the job market in software industry today, you know that there are plenty of opportunities for skilled developers. People can easily vote with their feet. From maintenance point of view this is a real pain.

New people, who weren't familiar with the codebase, had a big struggle getting into productive mode. Without really rigorous attention to architectural quality, software tends to get more complex and more difficult to maintain. And when this is combined with staff turnover, the overall architectural picture gets lost.

Unfortunately, when there are no senior members in the technical staff, the balance between business and technological needs gets disturbed. Business grieves features and visible things. For the business, development speed (how many new features you get with x amount of money) and performance (how quickly things happen when you click) are relevant. What happens behind the scenes is not. The development team should speak out for the technological needs.

There's a dependency between development speed and codebase complexity. With low automated test coverage and high coupling, introducing new changes is always a potential source of unwanted regression. But for a person observing the development from the outside, this simply appears as slow and costly development that produces poor quality.

Back to the story. The customer didn't want to spend so much on the maintenance and so the supplier scaled down the size of the development team. Or the team size was kept about the same, but they were working for another customer. Organizational changes happened on both sides, with the customer and with the supplier.

The customer was unhappy with the development results and felt that they needed to take a tighter control. Team was heavily micromanaged. Daily scrum meetings turned into reporting and command and control sessions. The development team didn't like this and it accelerated the staff turnover even more.

The Renewal

The supplier had seen that it was getting very difficult to find people who were familiar with the technology. Without consulting the customer, they made a plan how to renew the portal to current technology. The benefit would be that there was much more skillful labor available in the job market and that the development speed would increase dramatically. The software architecture had deteriorated into a big ball of mud, but by applying certain changes, the user interface could be separated.

Due to the understandable lack of trust, the customer had hired an external consultant to aid them in technological decisions. Actually this person was also one of the people who originally participated in making of the original portal. Thus he knew it well and didn't see a need for changes. The supplier was trying to push the idea, but things got postponed. After six months, the supplier had hired a technology guru who could fluently demonstrate how the changes could be implemented. It changed the customer's mind. The plans were the same, they were just presented smoothly by a charismatic person.

When the renewal project finally got green light, the customer insisted that there would be a "dream team" working on it. There would be team members from the supplier (let's call them team A) and team members from the other smaller company (let's call them team B). Architect and scrum master were appointed from the smaller company and the customer acted as the project manager. Customer strongly demanded that the development should take place on site, in team B's premises. 

Before the actual development started, there were planning meetings where the plan was discussed, argued and fought over. Architect from team B questioned all design decisions that came from team A. During this period both the technology guru and the most senior developer in this new technology resigned. They had made their decisions about how fruitful the co-operation between the companies would be and didn't want to live in continuous argument.

Team A had had a team lead who was the main contact between the customer and the supplier. But for this project the customer said that they didn't need his services, since the customer's own representative was managing the project. Team A also had an architect, but he was stripped of his power, since the achitect was appointed from team B. During all this time, team A was still contractually responsible for the portal.

Teams didn't collaborate that well and the project fell into crisis. Even though there had been numerous planning sessions before the project started (that were mostly used for arguing), the plan regarding the user interface was still rather abstract. Stories had not been refined and for example the styleguide for the whole site was only one story. The work estimates were too small, since they were given by more senior people than who were left in the team.

The End

The team lead, who wasn't needed for guiding the project, resigned. This wasn't communicated with the customer. Soon the whole renewal project was cancelled. The customer made a decision that they would end their contract with team A and team B would maintain the portal in the future.

Learnings

There are probably quite a few things to learn from this and mistakes were made in many occasions. But here are at least some takeaways:

• Pay attention to technical quality. Feature development is important, but without proper code and architectural caretaking software will rot over time.
• Technologies have lifecycle too. I dare you: try to find COBOL or Smalltalk developers. Make a plan for renewal early enough.
• You catch more flies with honey. If you want people (your own employees or suppliers) to walk the extra mile, treat them properly. Collaborate rather than command.
• Or from the supplier perspective: if you are faced with a challenging customer relationship, add a buffer role into the interface. Some people might even enjoy the challenge.
• Don't pit suppliers against each other. Rather insist that they collaborate.

End note: Hopefully the story made you think. If you see events around you that resemble this story, be careful. It's never too late to change the direction ...until it's too late.

Photos by Ariel Besagar, chuttersnap, frank mckenna, Matt Botsford & Filip Mroz on Unsplash

Value Stream Mapping SIPOC

I was this week in a workshop where we were introduced to a nice set of lean tools. 5 x why, fishbone and kano model were visited briefly, but most of the time we concentrated on value stream mapping. To share the knowledge I will tell you about it in more detail.

We used SIPOC model. The abbreviation comes from the words

• Suppliers
• Inputs
• Process
• Outputs
• Customer

In this example we were creating a breakfast toast for one of our family members. So customers in this case were husband/wife or child.

Outputs can differ if the different customers (or customer segments) have different needs. In our case the child wanted a toast with jam and special attention that the toast isn't burnt. Spouse wanted a cheese sandwich and all the facilities need to be cleaned. No crumbs.

Inputs are all the things we need in the beginning. In the toast example we need a plate, knife, jam, cheese, butter, bread (two different types) and heat.

All the inputs have a supplier. In this case we were thinking where in our kitchen they are located. Not really who made the knife. But the knife is in drawer, butter and jam in the fridge. Heat is provided by the toaster and plate in the cupboard.

Finally, when these pretty evident things are covered we can start to think about the process. WHAT activities are needed in order to create the desired outcomes? We first make a line of post-it notes that define the activities. We can label the different activities with different colors. In our case we used green for processing, yellow for transportation, blue for storage and red for inspection.

Each WHAT phase is broken down into HOW parts. For example toasting phase can be broken down into activities of setting the temperature, putting the bread into the toaster, setting the toaster on and the bread being heated.

We used heart shaped post-its to mark all the value adding phases. In any process there are activities that add value, activities that are necessary but not value adding and thirdly activities that are waste. By identifying these we can measure how much of our activities actually are value adding compared to the whole time used. This performance indicator is known as Value Added Ratio (VAR) or Process Cycle Efficiency (PCE).

PCE = Customer Value added time / Process Cycle Time

As in our example the spouse explicitly wanted to have no crumbs, we tracked all the parts where crumbs could be produced. These phases were connected with a string. This way we can only concentrate on improving those parts to minimize the amount of crumbs.

Finally it makes sense to identify your customer touchpoints. You can goof off in many parts (well, better not to too much..) but you should make sure that you handle the customer touchpoints with care. Think about what happens before, during and after the touchpoint. How does the customer feel?

There you go. A crash course to value stream mapping using SIPOC method. If you want to learn more, I encourage you to find a lean consultant or simply try it out. I know I will.

Leadership

A person starts to lead when she takes responsibility of something greater than just herself. Before this, she needs to learn to lead herself. This means she needs to take responsibility and make a decision to sit on the drivers seat. It is quite easy to blame someone else when something goes wrong. It takes courage to admit that I had the choice, I made the decision and it turned out to be wrong. But it's a necessary step towards leadership.

Leader should be vulnerable. Admit your mistakes. Say you are sorry. Ask other professionals for guidance and be willing to change your opinion when you see better facts and hear better arguments.

Don't try to be something you are not. Even if you have a few rough edges it might be better to show them than act. People can see through facades. But if you are rude, it still might be wise to polish your manners.

Empathy is a core leadership skill. But even empathy can be learned. Think about how your actions affect others. Or, better yet, ask them! And listen to the answers.

One dilemma in leadership is that many believe in leading by example. I do so too. But in expert organization that isn't usually a choice. The people in your team are better in their craft than you. So another leadership style kicks in: stay back and let other people shine. Support them. Clear obstacles from their path. And boost their confidence by telling them how much you respect them.

Inhale blame, exhale credit to others.

I'll open this up. You are the leader. You take the blame. Period. It doesn't really matter what happened, it can be discussed later (and should be analyzed with maybe 5xWhy?), but right there in the moment when shit hits the fan you take the blame. No excuses. Then, when the dust settles, you stop for a while with your team and have a retrospective. You learn and you try to make sure the same doesn't happen again.

And the credit part. When something great is achieved it is usually a group effort. Leaders many times get to represent others in events, but when positive feedback is given, make sure you remember that it is for the team. Thank them. Thank them in public. And celebrate the successes in private too. Raise toasts or have a bun together. (bun intended)

All pictures from Gratisography.

Resilience

I was recently in a one day refresher training about coaching leadership. About one and a half years ago the original training was longer, couple of months. It included lots of assignments and pair learning with my supervisor colleagues. The training was organized in house and participants came from different departments of my company. Actually, this was one the most powerful takeaways from the training: I got to meet and network with people who work in very different environments. But we all have something in common: leading people is our day job.

Our everyday realities differ. It's diffrerent being in sales or in the frontlines of customer service than in the R&D or doing software development project work. But in the end, for leader the goal is same: your job is to help your team members succeed.

Photo by jesse orrico on Unsplash

The training was implemented in such a way that there were numerous pair and group conversations. One topic was resilience, the ability to overcome a disruption (for example an organizational change). With a quick poll it seemed that almost all participants thought that they were very resilient. They were able to work under stressful conditions and stay sharp. Afterwards I began to think that

Is high resilience a precondition for a successful leader?

But which way does the causality work? Was it a precondition or had these people become more resilient after being in the management position? Maybe worth researching a bit more.

At least it would be a benefit. That in today's stormy waters (of corporate life) you can stay calm. But it's not only about you. Since even if you can get over things quickly, others might take longer. And if you forget this, you may move too fast.

• Give your people time to adapt. 
• Offer them opportunities to discuss and reflect. 
• And use your gift of keeping your head clear to help others.

Photo by Dane Deaner on Unsplash

Developers - Cost or Asset?

When things get big enough there might be functional silos that concentrate on specific activity. These could classically be something like sales, development, support and marketing. Of course these functions should be made as efficient as possible taking into account the whole. Customer value is usually built in co-operation between all the functions, not by a single unit.

Metrics can be a double-edged sword, if they are applied to functions instead of taking into account the whole value chain. Sales funnel is quite easy to quantify. You have numerable amount of possible clients. You can count how many units you can sell to this market, thus getting the absolute limit value for your cash inflow. It's easy to measure how much a single sales person or sales unit can close deals and bring in money.

If you sell development hours, it's also quite easy to measure how much a developer generates turnover or how big the billing rate is.

In typical cases salaries are even easier. There's usually a fixed sum of money that is paid to the person monthly. This is the cost of a developer (plus hardware & licenses too).

But it’s much more difficult to quantify the financial benefit of an in-house developer who works in R & D. One could make a rough estimate about how much an equivalent consultant would cost. But this doesn’t take into account the subtle knowledge that is accumulated over time or the internal relations. Possibly this person is the glue that keeps the team together, the humorist that keeps people in high spirits or the oil that lubricates the internal cogs and makes things happen. Not impossible, but difficult to quantify.

Without visibility and understanding, one can draw wrong conclusions about the situation. Classic would be rewarding sales department for bringing in the money and cutting costs in operations by sacking people.

 

"What should I do then?"

To remedy the silo problem and optimizing parts in expense of the whole, you should concentrate on metrics that measure the whole value chain and are difficult to gamble. Some examples:

• NPS. How the customer experienced your service?
• Lead time. How long it took your organization to meet the customer’s need?
• Profitability. Do not concentrate on (just) how much sales you make. Pay attention to how much it costs for you to develop, deliver and market the product/service.  

 

 Photos by "My Life Through A Lens", Spenser H, Fabian Grohs and Mike Wilson on Unsplash.

Employee Turnover and How To Avoid It

In the current market situation, software development professionals are the ultimate scarse resource. I'm not exaggerating if I say that these people, for example talented architects or data scientist, don't need to look for positions. They are being actively approached by headhunters and hiring managers.

They are neither easily replaceable, like cogs in a machine. Usually they possess knowledge that has been accumulated over a long period of time. Into some extent this information is useful in other environments, but it's most valuable in the current setup. That is why the existing employee is most valuable to their current company.

The cost of employee turnover

Losing a member from a well welded team has a significant cost that can be divided into (at least) the following parts:

1. Team loses a member and thus his/her direct contribution.
2. Team needs to reorganize and learn to function without the lost member.
3. Recruitment takes time away from development.
4. When the new team member is found, he/she needs to be oriented and inducted.
5. Team needs to reorganize again to function with the new member.

Sadly, I feel that many times people only think about the first bullet. And maybe if there's been difficult recruitments, also the third one. But all of these have a cost.

For the above mentioned reasons companies should do all possible efforts to keep their employees happy. I understand this is not possible for all company roles and it might be viewed as unfair. But it's really as simple as the law of demand and supply. With the rise of banks and other companies digitizing their services the demand is bigger than ever. (Almost) all companies are becoming software companies. And there's rather limited supply of software engineers.

Few tips on how to keep your employees

Compensation is one parameter in the puzzle. It should be on a decent level, but it isn't everything. Benefits that either have monetary value or make the life easier also count. I think the latter ones are more effective in retaining employees. Generally people are lazy (at least I am). Autonomy in choosing your own tools and ways of working increases happiness. If your work has a higher purpose or meaning to the society, it's a plus. For people who work with their brains, right amount of challenge is expected. And finally, you should be able learn new things.

There's a cost for keeping your employees, but usually it's smaller than losing them.