111

I made a code change (pull request (PR)) about six months ago that introduced a bug into our system. It ended up really causing a lot of issues and just recently it was brought to my attention. I tested the system with what it was before my code change and it worked fine. I made another PR that reverts my code change.

I just feel really bad about all of this and I don't think my supervisors are happy with it.

I have owned up to it and said this was my mistake and apologized to the team. However, I still feel bad.

What is the best way to approach the aftermath?

  • Should I privately apologize to one of my supervisors?
  • Should I just learn from my mistake and move on?
  • Should I do something else?
9
  • 9
    this looks related / duplicate: Should I send out an apology email or IM after introducing a production bug?
    – gnat
    Feb 9, 2022 at 17:14
  • 191
    So you didn't catch the error, but neither did your QA team nor 6 months worth of live usage. Why do you blame yourself?
    – Seth R
    Feb 9, 2022 at 19:39
  • 96
    In addition to QA not catching the bug, did your code get reviewed by other developers? If yes, apparently it wasn't obvious. If no, that would be a good addition to the company's processes.
    – Llewellyn
    Feb 9, 2022 at 20:05
  • 11
    @Llewellyn Yes it did. However, we have a bad habit of just blindly accepting PRs. I try to not be that way but yeah. I think our system is a little dysfunctional. I worked at a different company one time that actually did PRs the right way and the reviewers were just as responsible as the developer.
    – Slaknation
    Feb 9, 2022 at 21:22
  • 6
    @gnat No, this is not a duplicate. The post you linked is an entirely differet scenario. 1)In that case, it had escalated FAR beyond the team 2)post mortems had already been conducted 3)The CTO was involved 4)it had a far heavier impact 4)It was a critical application. 5)it is an entirely different scale. This is like comparing a minor leak in a rowboat to a hull breach on a battleship. xxxxxxxxxxx However, I did find a few words duplicated, such as "I", "Team" and "apologize". Other than that, however Feb 10, 2022 at 15:12

11 Answers 11

485
+100

I've been a Software Developer for over 20 years, and manager for at least 10.

Bugs do get introduced all the time. Did you introduce it on purpose? If not, there’s NO need to apologise. Only non-developers would expect it and that’s because they know nothing. You should not overthink it.

The team however should be thinking on how it happened. How did a critical bug get through all your code review, manual and automated QA process, and stay in for 6 whole months without anyone noticing?

You do not fix people, and you don’t prevent mistakes - because that’s impossible. Instead you set up processes so that when people make mistakes (and they will) there's a very high chance that mistake will be caught up and fixed before it causes havoc. And when even that is not enough (and it will not be), you learn from it and improve.

People make mistakes but the process is what's at fault when that happens, not the people themselves. If you have managers that don't recognise that, then that's a red flag.

4
  • 1
    Comments are not for extended discussion; this conversation has been moved to chat.
    – Kilisi
    Feb 12, 2022 at 5:00
  • "Did you introduce it on purpose? If not, there’s NO need to apologise." - Eh? I'm not sorry I ran over your dog madam...because I didn't do it on purpose!?
    – Fraser
    Feb 24, 2022 at 13:42
  • 1
    Your analogy is wrong and I can tell you're not a developer already. Devs will make mistakes it is part of the job, you can't be apologizing for something that is expected to happen every time it happens. A more accurate analogy would be a painter apologizing because they drop some paint on the floor everytime they do. You know they are going to drop some, they know they are going to drop some, so why the *** wasn't there any protection on the floor to begin with (you'll note there's always some protection by the way) Feb 24, 2022 at 14:02
  • In general I agree about "fix the process", however; there are times where the correct choice is do nothing because the expected cost of risking the same thing happening again is less than it would take to prevent that. Other times, the best choice is to take steps to tolerate failure; build your fault tolerance into production rather than the QA/release process.
    – BCS
    Feb 26, 2022 at 23:44
78

Ask for a "lessons learned" or a "post mortem" session. This is not just your mistake, but your team's from upstream to downstream.

The focus is not, and should not be focusing on blame, but...

  • What were the critical points of failure?
  • Remediation processes
  • How did this go undetected for so long?
  • How do we stop this from happening again?
  • Adjustment to processes to prevent this happening again.

Mistakes happen all the time in IT.

Some of my more famous screwups.

  • deleting an entire directory

  • wiping my own hard drive (it was much easier in the 1980s/1990s)

  • a major project that I spent months on, which had terminal scope creep

  • Repeated a bug in multiple projects, because I turned them all in before the first one was tested

The important thing is that I didn't lose my job for any one of those, and I shouldn't have.

Nor should you, nor should you even waste one second on worrying about it, or being ashamed.

A programmer is constantly learning, and part of learning is making mistakes.

After a mistake I made, my manager told me a story.

He had heard about an executive that made a huge mistake and cost the company about 10 million dollars. The executive wrote up his resignation and handed it to the CEO The conversation went like this:

CEO: What the hell is this?

Executive: My resignation.

CEO: I just paid ten million dollars for your training. Do you think I'm going to let that money go to waste? Get back to work, and don't bring this up again!

You just leveled up as a programmer.

You did everything right in handling the mistake:

  • You corrected your mistake
  • You admitted to your mistake
  • You are going to learn from your mistake.

Leave it be, don't dwell on it, take what you learned and grow as a programmer.

3
  • 4
    This. You can learn a lot about a company by looking at how they handle mistakes. If they make it about you and force you to apologise, walk away from them. If they consider it a reason to run a root cause analysis, fix their processes, QA, toolchain, whatever, stick with them. (note: exception for cultural reasons, in some cultures it is simply customary to apologise for mistakes)
    – Tom
    Feb 10, 2022 at 6:05
  • 1
    @Tom: Thanks for the exception. I've never been asked to apologize, but I typically apologize to the people who were affected, and thank the people who helped me. I used to be in a team where the custom was to bring croissants to the team the next morning; nothing better to banish bad feelings towards someone than to be offerred a fresh, still warm, croissant for breakfast :) Feb 10, 2022 at 14:48
  • 1
    "wiping my own hard drive" That's super easy to do when repartitioning on a Mac. It doesn't ask for confirmation or delay the process for when you end the utility, it just does it, immediately. It's the partition for the running OS? Eh, whatever, it's gone. You mis-clicked? Too bad and too late, it's already done. Feb 10, 2022 at 16:23
31

You should not overthink this. You're not the 1st developer to introduce a bug in a system, and certainly not the last.

Learn from this mistake, take the necessary steps to make sure this won't happen again (or to reduce the risks of this happening again), and move on.

5
  • 8
    „and certainly not the last” You don't know when society will collapse
    – Thomas
    Feb 10, 2022 at 9:01
  • 5
    @Thomas If society collapses, there will still be a lone developer in a windowless office in the basement trying to figure out why his API calls are being refused right up until the internet to the building shuts down. Feb 10, 2022 at 14:14
  • 5
    @Thomas, I'm quite sure that new bugs have been introduced to software since the OP made their mistake, and even since they posted this question. Feb 10, 2022 at 16:24
  • @Ruadhan2300 One of the final chapters of On The Beach has some characters planting out bulbs for the next year, in the certain knowledge that the entire human race will be dead before Spring. That turn of mind goes double for software developers.
    – Graham
    Feb 10, 2022 at 19:33
  • @Graham "one day our children will dig up my code and it will be the last remnant of a dead society. I'd better make it good!" Feb 11, 2022 at 8:31
9

When software engineering was in its infancy, many scientists believed it should follow the path of formal engineering and establish a discipline of delivering software based on formal methods and mathematical proofs. This would ensure bugfree software to be developed, provably so.

However, outside of a few mission-critical applications, this approach is prohibitively expensive and time-consuming. In practice, the industry has settled on a trade-off using some mitigations like test-driven development (TDD), QA processes and code reviews vs. getting code out to production faster.

This means that it is impossible to be sure that any code we deliver is bugfree. The best we can do is to show we have used the tools at our disposal to minimise the probability of introducing new bugs. TDD and code reviews catch a lot, but they are limited tools exposed to human error, and requires a feedback loop to improve when errors do make it through.

The bottom line is that bugs will happen and it's not always anyone's fault. From this question it is clear that your attitude is spot on: analyse why it happened, how it evaded the quality mitigations, and try to introduce new mitigations to prevent this from happening again. This is part of the job.

1
  • 2
    Comments are not for extended discussion; this conversation has been moved to chat.
    – Kilisi
    Feb 12, 2022 at 4:59
4

Do you think it would be a good idea to go to one of my supevisors in private and apologize?

You have already publicly apologized, there is no need to privately apologize to your supervisor unless they were not present for your public apology.

Or should I just learn from my mistake and move on?

Yes, make sure you understand exactly what you did wrong and make the necessary adjustments to prevent it from happening in the future. If your company does not already have a group dedicated to testing changes, this is something you can bring up to improve the impact of any future changes.

5
  • Ok. Thanks. Yeah my company does have a team that is supposed to test changes so I think they are also partially in the wrong but I was the one that introduced the change so I take responsibility.
    – Slaknation
    Feb 9, 2022 at 16:06
  • Joe, I had one bug once that crashed your machine if you put it to deep sleep and woke it up after 35 to 45 seconds. That was due to a totally unnecessary change by a very senior developer who didn’t think anyone should review his code. And the change did exactly the opposite of what was intended.
    – gnasher729
    Feb 9, 2022 at 19:36
  • @JoeStrazzere No they don't. Our team is horrible about PRs. They just accept without hardly looking at the code.
    – Slaknation
    Feb 9, 2022 at 19:53
  • @Slaknation Oooooo, that needs to change. No matter how good a programmer you are, you always miss your own mistakes. Feb 10, 2022 at 21:57
  • "...make sure you...make the necessary adjustments to prevent it from happening in the future." The bug was apparently not even noticed for six months. That's a good indicator that bad process is a serious problem in that shop. From the sound of it, the developer is relatively junior and does not have the power to push through the kind of sweeping process changes that would be needed to prevent this from happening in the future. He may not even be able to prevent himself from doing it again if the system is large given that it's obviously lacking in test coverage for things like this.
    – cjs
    Feb 11, 2022 at 4:56
0

Firstly, consider that commercial software vendors do not typically issue personal apologies to users when they push out upgrades due to finding a regression. If you apologize for making a bug, you've gone beyond the industry norm.

A software team shares the responsibility for all changes. Or if they don't they should. There should be a peer review process for every commit, and tests to catch regressions.

The regression you caused slipped in because:

  • It was not caught in the peer review (was there a peer review)?

  • Whatever behavior the change broke was not covered by a test.

If you are working some feature X, and some feature Y breaks without being detected, it's not entirely your fault. It's partly because the behavior of Y is not covered by a regression test suite.

(In addition to reverting your commit, you should add a test case somewhere which reliably passes when your bad commit is absent and reliably fails when it is present.)

Of course, you're responsible for tracing all the ways which your change can affect existing logic, but the less detailed is your regression suite, the more time-consuming is that tracing process, and the more likely you are to miss something in spite of a good effort.

Also, it may be outright impossible to avoid breaking some behavior that is neither documented nor covered by a test. Why? Because a behavior that is not documented or test simply does not exist as far as you're concerned.

Some other developer knows about the behavior, and then several months or years down the line remembers it and tries it. Then you get blamed: hey you broke this Y thing with your X change back in 2014! Except Y thing is only documentedin that developer's head, and not covered by a test.

Never accept responsibility for "breaking" something that was never documented or covered by a test but expected by someone to work. Call the situation out.

Sometimes the downstream users rely on undocumented, untested behaviors. Things can break that way; you ship a new version and the users complain that something that used to work doesn't work. Still, do accept some personal responsibility for something like that. It's nobody's fault. Every complex system ends up with behaviors that are not documented. Only very technical users (engineers) understand the implications of discovering some "feature" and relying on it without checking that it's actually a feature. Ordinary users assume everything that seems to work is a feature. If you break something like that, the blame is distributed: the users should have known not to rely on something undocumented; your software, overall, could have been better in validating against unexpected input combinations, and checking internal contracts, so that there are fewer such undocumented behaviors (ideally none); or else, your team, including QA people, could have been better at identifying such undocumented behaviors, and communicating them to documentation people so they could warn the users.

0

"But... did you DIE?"

Lighten up, friend!

Even sitting presidents of the most powerful nation in the world find themselves messing up royally. Stuff happens. There's no need for you to cower in a corner somewhere.

Be thankful that there's enough infrastructure in the environment that you work in such that mistakes you make aren't causing your company to be calling you at 2AM twice a week. QA and first-line support teams are your best friends when you're working as a developer, and every time I interview, I make sure that I'm have this kind of support. Even with QA and unit testing, bugs will slip through.

The Java runtime license terms specifically decline any liability for code running in aircraft operations or nuclear facilities because someone high up understands that stuff happens.

I own this shirt, from Six Dollar Tees:

one of my favorite t-shirts

2
  • I have a relation who's a doctor and "Yeah, but did anyone die?" is basically the motto of their profession right now. When they say it they really mean it too.
    – JeffUK
    Feb 11, 2022 at 11:49
  • 1
    "Even sitting presidents of the most powerful nation in the world find themselves messing up royally. " - and also, none of those people got where they are by then apologising for said messes! Pfft!
    – mgraham
    Feb 11, 2022 at 12:32
0

Did you bypass any of the team's processes?

Unless you bypassed normal process for your team (e.g. you worked directly on master, or you deployed to production without submitting your code for Q/A), or unless you otherwise violated a company policy or some law, it really isn't your fault--as others have said 1) the process of writing code introduces bugs as a fact of life, and 2) if bugs make it into production it is a process failure, not a software development failure.

It's most likely not your fault--it's the team's

So if you didn't bypass this process or do something far outside the norms of software development it's not your fault, don't beat yourself up.

If you're getting blamed and it's not your fault, start looking elsewhere

And if you're getting blamed for it by your management, you might want to consider brushing up your resume and finding a better (less toxic) place to work.

0

Look at the big giants like Google, Microsoft, Amazon etc, how frequently they bring the fixes. All programmers make mistakes and they are subsequently rectified (even a perfectly running code may have severe security flaws). However, searching a scape goat to save the face in front of the client, is an example of bad leadership and organisation culture.

-1

You've done what's needed...owned up, fixed it, accepted responsibility, etc.

Now you own it. Remember what led up to it, each place you could have avoided it, and the thought process behind each mistake. When you see others in similar situations, relate the anecdote as a dark warning.

This not only leads to self-improvement, but leaves the impression that that was the only mistake you ever made in your life. :-)

-1

The other answers all show that so far you have done the correct thing and you need some sort of review to see how it happened.

However given in your question you say

I tested the system with what it was before my code change and it worked fine.

Any review has got to look at other issues.

If your test showed it worked at the time - then something later showed up or created the bug.

Was your test added to the testing scenarios at the time. If not then why was this not done your team needs to make sure all tests are added, in fact someone else should have run this test before the PR was merged. If it was added then was the test run after every other change if it had been it would have caught the change that exposed the bug, if not why was it not run.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .