67

I'm a senior developer in a mid-sized business. We are split into multiple sub-teams with specific roles for each. My allocated team's responsibility is for the day to day running of services and bug fixing issues that arise from our product.

I have recently had a bad case of burnout related to the current COVID situation. Amongst the several concerns that I have are that of an individual team member. This team member has a habit of making breaking changes, without much thought or consideration given to others in the team, let alone across sub-teams. It is almost the bane of my life, multiple system outages have been caused by this person, but they continue at an astonishing pace of development. This is obviously viewed by management as a "positive" thing as they're getting the work done.

The catch here is that each time there is an outage, the investigation follows. This involves multiple meetings, bug fixing (my team) and slowdown in our work while we put out the inevitable fires that have been started. Worst of all, it is really hard to get them to own up to their mistakes. Talking with them, it's never "their" code that's at fault. The individual even tried to blame the hardware ECC memory for memory corruption that originated from this individual's code.

I am all for people learning from mistakes, but we barely get to put out the last fire before the next one is started. Most recently, and while I have been under a lot of pressure to deliver, I went to get the final task of the day done. Lo and behold, they are at it again and a code commit broke my work.

I quite publicly flipped out on our team channels about it because I was pretty fed-up with the direct approach. This has got management attention, in that their manager got involved. After talking with management time and again, little seems to have changed with this individual.

How do I raise this to management’s attention, get them to act and, more importantly, help prevent this from continuing?

  • 69
    Does your team do code reviews, merge requests, etc.? What kind of quality control is in place for this product? – GB1553 Jul 16 at 17:56
  • 7
    @GB1553 there is a code review process, but the range and scope of the changes are sometimes quite large. I feel this might be part of the issue as people treat the code review process as a tick box excercise rather than taking the time to properly assess the code. – user119822 Jul 16 at 18:08
  • 14
    How do code reviews which are treated as formalities and bugs that require a specific set of untested circumstances constitute a "rogue" developer? – buckminst Jul 16 at 19:25
  • 2
    @buckminst when that individual is the source of a continual stream of failures, and each of those failures point to a source of low quality / half checked work. Are we to say that the quality of ones code entirely depends on the code reviewer? I don't believe that should be the case. – user119822 Jul 16 at 22:31
  • 2
    @user9237382 - Do you have automated tests ? Can they be run via CI server like Jenkins ? Do you run regression regularly ? Repository hosting sites have options to automatically prevent pull requests until at least 1 build/regression test has passed. Besides, save the Jenkins builds so that you have history which can be used as evidence if needed. – MasterJoe Jul 18 at 5:59
145

You must make an ironclad rule that if a person commits code that breaks the build, the commit is instantly reverted and that person's work is not completed.

If you don't have the authority to make and enforce such a rule, then you must convince the party who does have that authority. Document completely the actual costs of breaking the build.

Until this rule is in place, act as if it were. When the build is broken, revert to the last good build. Demonstrate that the only difference between project software that works and project software that does not work is that one commit.

You should be making a lot of noise about this, to everyone from your colleagues to upper management.

| improve this answer | |
  • 61
    Greg is correct. No one cares if you are inconvenienced or annoyed. The point to hammer is broken builds cost money. – A. I. Breveleri Jul 16 at 18:13
  • 3
    I think this is sound advice, some of the issues around this is that builds break and they are not picked up on for a period of time. My particular team is a little more sensitive to this as we tend to pickup on build notifications more often. Because of the use of shared components, I'd like to treat those more as a contract (much like consuming a package) and require said team to publish their changes for components that consume them. Their build breaking, or implementing breaking change shouldnt effect us so badly? – user119822 Jul 16 at 18:15
  • 2
    Sorry, I thought you were talking about a member of your own team. The problem is different if it arises from another team. – A. I. Breveleri Jul 16 at 18:31
  • 16
    @user9237382 “builds break and they are not picked up on for a period of time.” — this is a major factor, if not the real underlying problem. The sooner you can spot a broken build, the more obvious the cause of the breakage will be, and the easier this will be to handle. If you had automated tests, that could be very quick — and if your build or revision control system ran them as part of the check-in process, your colleague wouldn't even be able to check in broken code! – gidds Jul 17 at 8:55
  • 4
    This answer assumes that the official culture of the company is not 'move fast and break things', It is really not up to the OP to decide what the right balance between progress and stability is; it is really a question upper management needs to answer. If the management is fine with sometime broken software but fast progress (like it seems since they praise the developer) he might be up for a reprimand if he follows your answer. – lalala Jul 17 at 9:13
46

With respect to getting management to act it's helpful to quantify the problem in relevant terms (money is best, time is also acceptable).

For example:

Change 123 broke the system, resulting in X hours of downtime and costing us $Y in lost revenue.

Change 456 created bugs that upset Big Customer (costing them $ etc.). Big Customer is now reconsidering their relationship with our company.

Change 789 caused an outage which required Z hours of development time to fix. That time came from Important Project X. Important Project X is now delayed Z hours.

The idea is to frame the issue as one impacting the business.

As such it's equally important when writing such a list to ignore things that aren't impacting the business (directly). It doesn't matter how fed-up you are (sorry), it doesn't really matter that you're having to work a lot (again, sorry). It also doesn't matter if this particular individual hasn't responded to your previous attempts at getting them in line. None of these are really business problems.

As to what your managers ought to do (and you should be ready with suggestions) it depends on what your company already has in place. If there are supposed to be code reviews prior to commit for example, but they're not happening, or they're not being done effectively, that could be addressed. If you don't have code reviews you could suggest starting them (of course they cost time/resources too so that has to be balanced). You could implement various testing regimens, or you could actually use the ones you do have (if you have some) etc etc.

| improve this answer | |
  • I think this is helpful too, by using quality gates as a direct form of control for some of these behaviours it might allow us to get a better handle on it. This individual has been known to do the "just because I can then I will" approach, which has resulted in some rules already being added. – user119822 Jul 16 at 18:18
  • 7
    In re "It doesn't really matter that you're having to work a lot"--to the extent this is taking development effort away from prioritizing OP's actual deliverables, contributing to burnout, and might result in senior staff turnover [which, OP: consider!], this shouldn't just be casually brushed off. And it sounds like it's requiring overtime and repair-work from an entire team. That has to be made visible to counterbalance the false appearance of productivity of the busybody who's making a career out of piling up technical debt. – Tiercelet Jul 17 at 14:19
  • 6
    As a piece of rhetoric, it can be useful to focus on money/business bottom-line impacts. But to say the other stuff "doesn't really matter" is just false. E.g., it's well known that money alone is not the best motivator of employees. If the OP's well-being is sufficiently degraded they (and other developers) should leave, and then the business is in a death spiral. – Daniel R. Collins Jul 17 at 14:39
  • While employee-turnover induced death spirals and whatnot certainly can and do happen there's nothing in this question to indicate it's happening at this particular company. There's an issue impacting one of what OP called "multiple sub-teams". In that context management will be more inclined to act on business issues. Communicating the (apparently sub-quitting level) dissatisfaction of one person, or even one sub-team of persons is unlikely to spur the changes OP is looking for and as such doesn't matter in the context of this particular conversation. – Greg Jul 17 at 15:17
11

It's their manager's responsibility to deal with all "personnel issues." You should speak to your manager, describing the situation, so that (s)he can speak to the other manager.

You should be fully prepared, of course, for them to also want to speak to you, and you, too, are obliged to do your part (as they see it) to achieve resolution. Be prepared to be told -- and, to accept -- that you may be completely or partially wrong. "Be professional."

| improve this answer | |
  • 1
    Thanks for this. What I want to be overall is "professional" about it. I acknowledge this is a two way street and that my involvement with said individual has been less than what I'd expect of myself already, it's certainly something I'll raise. – user119822 Jul 16 at 18:11
9

Perform retrospectives and iterate your process to be resilient in the face of one person's error.

Performing blameless incident retrospectives and regular development team retrospectives where you focus on the issues, all learn about your system, and devise ways to make your system resistant to errors is the best first step.

Identify the issues, figure out frequency and cost metrics around them, and figure out how to make it better. Does there need to be more testing and higher code coverage? Do people need to be committing more frequently so the PRs are smaller? Do more people need to be on a PR? Are build failure notifications being sent in actionable ways (in chat, in email, etc.) to the right party?

A developer should be free to make mistakes. There should be a framework in place that helps all the devs find errors, quickly, before passing them down the line.

Maybe there still needs to be personnel action - maybe.

Some folks believe "everything" is the system's issue. In reality there are some people who are slackers or screwups and no reasonable amount of development guardrails will correct that.

However, that's rarer than people think. If you have a very prolific developer that creates 5x more code than their colleagues, then if they create 5x the bugs/incidents of their colleagues that's normal, and I would prefer that employee to other employees because I'm getting 5x the output. It's only if they are causing 10x the problems that calculus starts to cross back over the line.

Even if you need to take this to their manager, be focused on solutions. "Talking to" the manager and their manager "talking to" them hasn't worked and usually doesn't. Engage him in fixing the problem. "How can flaws in your team's work not affect ours? What agreements or technical safeguards can we put in place to make this best for everyone?" Keep in mind YOU should not be doing this, but your manager should be talking with the other team's manager.

| improve this answer | |
  • 8
    What you're neglecting in your 5x analysis is developers like the OP described aren't fixing 5x the problems. Their productivity is only gained by leaching off of everyone else's productivity who has to clean up the mess. Otherwise, why not tell your entire department to code like that guy? – Karl Bielefeldt Jul 17 at 17:41
  • Retros are all the rage these days and I think they make sense. However the execution of a retro and making sure they are called regularly rather than when there is an issue, determine whether it helps or just injects more politics into the environment. – Mark Rogers Jul 17 at 21:29
  • Well, also the process should get the team involved in supporting/ fixing bugs in their own code. And yes, you need to carefully follow a blameless/blame-aware retro format to keep things focused (plenty of info available at the link I added and/or googling). – mxyzplk - SE stop being evil Jul 18 at 17:00
  • I define retrospectives as regularly scheduled meetings. If they only occur when there's a problem, they are as best post mortems and at worst finger pointing blame games. Process improvement vs fire-fighting. – Michael Durrant Jul 18 at 19:55
  • There are two kinds of retrospective, incident retrospectives and team/sprint retrospectives. Both are helpful. – mxyzplk - SE stop being evil Jul 18 at 20:12
0

I agree with others who've advocated using a CI system and tests. In my startup, we run tests against each pull request, and only those that pass can be merged. After the merge, tests must pass again before the latest merge is deployed to production. And if a pull request breaks production, it is immediately reverted. Then it's up to the developer to clean it up, get it through code review again (usually with new tests to demonstrate that the problem is fixed), and ensure it passes tests. We still have problems that take down production, but only for a matter of minutes at most. If in doubt, we revert first and ask questions later.

| improve this answer | |

You must log in to answer this question.