How do I deal with a team member who regularly makes breaking changes?

Question

I'm a senior developer in a mid-sized business. We are split into multiple sub-teams with specific roles for each. My allocated team's responsibility is for the day to day running of services and bug fixing issues that arise from our product.

I have recently had a bad case of burnout related to the current COVID situation. Amongst the several concerns that I have are that of an individual team member. This team member has a habit of making breaking changes, without much thought or consideration given to others in the team, let alone across sub-teams. It is almost the bane of my life, multiple system outages have been caused by this person, but they continue at an astonishing pace of development. This is obviously viewed by management as a "positive" thing as they're getting the work done.

The catch here is that each time there is an outage, the investigation follows. This involves multiple meetings, bug fixing (my team) and slowdown in our work while we put out the inevitable fires that have been started. Worst of all, it is really hard to get them to own up to their mistakes. Talking with them, it's never "their" code that's at fault. The individual even tried to blame the hardware ECC memory for memory corruption that originated from this individual's code.

I am all for people learning from mistakes, but we barely get to put out the last fire before the next one is started. Most recently, and while I have been under a lot of pressure to deliver, I went to get the final task of the day done. Lo and behold, they are at it again and a code commit broke my work.

I quite publicly flipped out on our team channels about it because I was pretty fed-up with the direct approach. This has got management attention, in that their manager got involved. After talking with management time and again, little seems to have changed with this individual.

How do I raise this to management’s attention, get them to act and, more importantly, help prevent this from continuing?

Answer 1

You must make an ironclad rule that if a person commits code that breaks the build, the commit is instantly reverted and that person's work is not completed.

If you don't have the authority to make and enforce such a rule, then you must convince the party who does have that authority. Document completely the actual costs of breaking the build.

Until this rule is in place, act as if it were. When the build is broken, revert to the last good build. Demonstrate that the only difference between project software that works and project software that does not work is that one commit.

You should be making a lot of noise about this, to everyone from your colleagues to upper management.

Answer 2

With respect to getting management to act it's helpful to quantify the problem in relevant terms (money is best, time is also acceptable).

For example:

Change 123 broke the system, resulting in X hours of downtime and costing us $Y in lost revenue.

Change 456 created bugs that upset Big Customer (costing them $ etc.). Big Customer is now reconsidering their relationship with our company.

Change 789 caused an outage which required Z hours of development time to fix. That time came from Important Project X. Important Project X is now delayed Z hours.

The idea is to frame the issue as one impacting the business.

As such it's equally important when writing such a list to ignore things that aren't impacting the business (directly). It doesn't matter how fed-up you are (sorry), it doesn't really matter that you're having to work a lot (again, sorry). It also doesn't matter if this particular individual hasn't responded to your previous attempts at getting them in line. None of these are really business problems.

As to what your managers ought to do (and you should be ready with suggestions) it depends on what your company already has in place. If there are supposed to be code reviews prior to commit for example, but they're not happening, or they're not being done effectively, that could be addressed. If you don't have code reviews you could suggest starting them (of course they cost time/resources too so that has to be balanced). You could implement various testing regimens, or you could actually use the ones you do have (if you have some) etc etc.

Answer 3

It's their manager's responsibility to deal with all "personnel issues." You should speak to your manager, describing the situation, so that (s)he can speak to the other manager.

You should be fully prepared, of course, for them to also want to speak to you, and you, too, are obliged to do your part (as they see it) to achieve resolution. Be prepared to be told -- and, to accept -- that you may be completely or partially wrong. "Be professional."

Answer 4

Perform retrospectives and iterate your process to be resilient in the face of one person's error.

Performing blameless incident retrospectives and regular development team retrospectives where you focus on the issues, all learn about your system, and devise ways to make your system resistant to errors is the best first step.

Identify the issues, figure out frequency and cost metrics around them, and figure out how to make it better. Does there need to be more testing and higher code coverage? Do people need to be committing more frequently so the PRs are smaller? Do more people need to be on a PR? Are build failure notifications being sent in actionable ways (in chat, in email, etc.) to the right party?

A developer should be free to make mistakes. There should be a framework in place that helps all the devs find errors, quickly, before passing them down the line.

Maybe there still needs to be personnel action - maybe.

Some folks believe "everything" is the system's issue. In reality there are some people who are slackers or screwups and no reasonable amount of development guardrails will correct that.

However, that's rarer than people think. If you have a very prolific developer that creates 5x more code than their colleagues, then if they create 5x the bugs/incidents of their colleagues that's normal, and I would prefer that employee to other employees because I'm getting 5x the output. It's only if they are causing 10x the problems that calculus starts to cross back over the line.

Even if you need to take this to their manager, be focused on solutions. "Talking to" the manager and their manager "talking to" them hasn't worked and usually doesn't. Engage him in fixing the problem. "How can flaws in your team's work not affect ours? What agreements or technical safeguards can we put in place to make this best for everyone?" Keep in mind YOU should not be doing this, but your manager should be talking with the other team's manager.

Answer 5

I agree with others who've advocated using a CI system and tests. In my startup, we run tests against each pull request, and only those that pass can be merged. After the merge, tests must pass again before the latest merge is deployed to production. And if a pull request breaks production, it is immediately reverted. Then it's up to the developer to clean it up, get it through code review again (usually with new tests to demonstrate that the problem is fixed), and ensure it passes tests. We still have problems that take down production, but only for a matter of minutes at most. If in doubt, we revert first and ask questions later.