71

I deployed a new version to a critical application in production last week. Post deployment we found out that the code in question ran into an edge case and we had to quickly roll back to the previous deployment. We ran the buggy code in question through multiple automated/manual tests in lower risk environments for more than a week before pushing it to production. Unfortunately, it was discovered only in production.

Now, I work in a large tech organization with multiple levels of engineering directors reaching up to the chief technology officer. Even though the deployment was reverted pretty quickly, the issue has now been escalated to higher management. We've gone through multiple root cause analysis meetings since then with multiple audiences and I've made sure to acknowledge I was the one who introduced the particular code which had the bug. I feel embarrassed for not recognising the issue in my code well in time and causing a lot of hassle for my team. However, fixing the issue, and fixing our process around deployments to ensure this does not happen again, has been a great learning experience for me as an engineer.

Should I send out an apology email/IM to my team as well? To apologise for all the hassle caused to them because of this? I don't want to come off as cocky/unprofessional in this case.

14
  • 32
    Who else was involved in the decision to push directly to production, or was it your decision and yours alone? Who (if anybody) reviewed your code before pushing?
    – Jeroen
    Jul 12, 2021 at 12:10
  • 80
    A bug in production is a test case that wasn't written yet...
    – Laurent S.
    Jul 12, 2021 at 17:04
  • 76
    Congratulations, you’re a productive software developer. Jul 13, 2021 at 1:02
  • 7
    Do you plan to write an email expressing your pride at every bug fix as well ? Because that's the (absurd) logical next step. No, you don't take personal responsibility because, heads-up, your bosses won't take personal responsibility for their mistakes - they'll usually blame someone else, possibly you even when you did nothing wrong. Jul 13, 2021 at 3:55
  • 30
    The fact that it escalated to higher management indicates that they correctly recognised it as a systemic issue, rather than one person's mistake. If it was just your mistake, it probably wouldn't be escalated beyond your immediate manager.
    – BenM
    Jul 13, 2021 at 4:15

10 Answers 10

248

Mistakes happen to all of us. The key thing is to understand them, learn from them, and avoid them the next time.

According to your question, you and the company did all the steps needed. On top of that, you also acknowledge your error

I've made sure to acknowledge I was the one who introduced the particular code which had the bug

I feel that an apology email / IM is too much. You already took responsibility over the issue. You can stress your embarrassment to your manager on your next 1:1, stating what you learned from the incident.

5
  • 199
    In addition, the key issue is not that code had bugs. The issue is why weren’t those bugs caught before they got put into production. A system where one developer has to shoulder all of the blame for a production issue is broken.
    – ColleenV
    Jul 12, 2021 at 14:31
  • 5
    @ColleenV There should be a separate team that focuses on production deployment / testing / QA. For a single developer to handle a mission critical production deployment is just a recipe for disaster. In a 6 man shop I worked at, we mandated 2 full time staff monitoring major system wide deployments, and 1 full time staff for client specific systems. Due to a past deployment blowing up over a weekend, we also did not allow deployments past Wednesday.
    – Nelson
    Jul 13, 2021 at 1:58
  • @Ozil These are wise words in this answer. If you need to say something beyond the acknowledgement in some future meeting, I would suggest that you don't continue to apologize, but instead tell your audience how you now alter your behavior to avoid the scenario (by writing more automated unit tests, preferably).
    – Edwin Buck
    Jul 13, 2021 at 4:37
  • 1
    @ColleenV exactly. Any code is bound to have bugs, and thats ok.
    – Hobbamok
    Jul 13, 2021 at 10:08
  • 4
    @Nelson it's not necessary for deployment, testing or QA to be handled by another team, and indeed it's increasingly common for experts on these matters to be embedded within teams, rather than a separate team. The wider point, that it is vitally important that experts on these matters are involved, is absolutely spot on.
    – James_pic
    Jul 14, 2021 at 10:49
128

No, this was the team's mistake, not just yours

It sounds like where you work has a multi-step process to get changes to production - that's good. It means that any mistake passed review by multiple people. Yes, you made a mistake, but so did the other engineers and QA, who didn't test for this either. Anyone could have made that mistake, and it just happened to be you.

We've gone through multiple root cause analysis meetings since then with multiple audiences and I've made sure to acknowledge I was the one who introduced the particular code which had the bug.

You already did what you needed to do - took responsibility that it was your code that caused the problem, and participated in meetings to change the process so this won't happen again. They realized there was a problem and reverted - that's the process if something goes wrong in production - the system is working. Every engineer has at least one story like this, and it's a normal part of software development.

6
  • 18
    They do say that someone who makes no mistakes, makes nothing.  So yes, we've all been there…  What matters is how you deal with it: rolling back quickly, analysing and fixing the problem, and admitting responsibility are all good things to have done.
    – gidds
    Jul 12, 2021 at 22:46
  • 5
    But OP's only fault was writing the dodgy code; the fact that it got into production is the fault of their team, their department, and to an extent the whole company.  Because people do make mistakes, companies should be set up to cope with problems; that's why we have several levels of reviewing and testing.  And yes, occasionally all that is not enough — that's how you learn to improve the process!
    – gidds
    Jul 12, 2021 at 22:47
  • Many good answers. This one is great because it highlights that code is owned by the team, not an individual. Responsibility is shared.
    – Alex L
    Jul 13, 2021 at 13:46
  • The bold text here is the key - the fault was not that the bug was created but that it made it to production.
    – Techlead
    Jul 14, 2021 at 8:35
  • Chances are also that some context made the error more likely (poor definition of use cases/limits, poor library/module interface documentation etc.). If all edge cases are determined and documented they are less likely to be overlooked ;-). Jul 15, 2021 at 11:22
20

No, absolutely not.

If your development process is such, that a single person's mistake causes a problem in production, then you have a problem with your process. Focus on getting that general problem resolved, rather than obsessing over the individual bug.

The last thing you want to do is to contribute to a culture in which blame for production bugs gets assigned to individual developers. That just creates a toxic environment for everyone.

9

These things happen. Sooner or later some more critical bugs make it into production. All these meetings and hassle that you and your team have gone through is not about a blame game like who made it or who didn't test this, etc. The reason for all meetings and so on is all to learn from and see how it can be prevented from happening in the future. Do you guys need some more automated tests, add some test case or change anything in the delivery process?

It is all just about increasing quality of the product. You have learnt from this which is the best outcome and no apology is needed. I would have been more worried if these meetings and root cause analysis didn't happen. It shows that you are in a team that wants to improve and cares about the product.

4

Getting and reading an unnecessary mail is an annoyance for most people. A mail that interrupts 12 people's workflow for only 5 minutes each also costs the company an hour worth of work.

If possible during corona-times: Consider putting a cake in the office-kitchen with a "sorry for the bug"-note. Everyone likes that and it does not cost the company. Even better: Instead of being "the guy/gall who causes dangerous bugs", you will be perceived as "the guy/gall who causes so few bugs that they actually bought cake for everyone the one time it happened".

5
  • 5
    Make the cake look like a bug :D (but something nice, like a ladybug)
    – Michael
    Jul 12, 2021 at 20:31
  • 1
    @Michael The original bug (found in the Mark II by Grace Hopper, et. al.) was a moth. While it was probably one of the boring brown ones, artistic license would certainly allow one that looked more like a butterfly (e.g. a cecropia or rosy maple).
    – mdfst13
    Jul 12, 2021 at 23:55
  • 1
    I've been there and done that and the cake was greatly appreciated. It also set a precedent, I'm glad to say.
    – RedSonja
    Jul 13, 2021 at 9:17
  • @mdfst13 the term "bug" predates the moth. It was engineering term for "small error" with any kind of thing. The page in the log said "First actual case of bug being found" was pun based on the already known usage.
    – VLAZ
    Jul 13, 2021 at 10:59
  • @Michael - even better, make the cake in the shape of Acherontia styx and leave a nice bottle of Chianti (more people have seen the movie than read the book) next to it. End result? You get to keep the cake for yourself as literally no-one will risk eating it :-)
    – Spratty
    Jul 13, 2021 at 11:28
3

Other people have already said why you shouldn't. I'm going to add another reason...

Exemplify what you'd expect from other people in your team/organisation

Do you expect the rest of your team to send out "mea culpa" emails when they screw up? I certainly hope not.

What you should expect is for them to own the issue when it comes to analysing the problem, to not try to find excuses, and to do their best to resolve it. That's what professionalism looks like. Sort the problem, wherever possible make sure it can't happen again, and move on. That's what you'd expect from them, so do the same yourself. It's important that everyone has a dispassionate approach to this, because making it personal is a toxic attitude.

Even if this is customer-visible, the person sending out apologies to customers should be management-level. And they should be taking collective responsibility as a company for the fault, not pointing the finger at any one person.

Always remember Maxim 70: "Failure is not an option - it is mandatory. The option is whether or not to let failure be the last thing you do." Sooner or later, something is always going to go wrong. What defines you and your organisation as professionals is how you deal with failure.

1
  • +100 if I could.
    – bob
    Jul 15, 2021 at 13:07
3

What would such an email achieve? The company and its employees should be concerned on how to reduce error rates. If you can propose an effective way of preventing this type of bugs in the future please go ahead. Playing the blame game is non productive even if you are blaming yourself.

That being said, it is good to orally acknowledge mistakes in a casual talk with immediate co-workers. Not only does it help foster team work, but you can also share knowledge and experience with your peers so that you can prevent similar events in the future.

1

Thinking about Jim Carey in the movie, "Liar, Liar" right now. 🤡

Don't deny that you made a mistake, but otherwise just let it go. This is software engineering, and it is an extremely complex task. Everybody knows that everybody makes mistakes. Hell, you can't do this thing without making mistakes ... which is why someone invented the term: "D'oh!" 🤦‍♂️

Now, this is also a good time for a little "process-improvement introspection." How did this happen? How did this problem make it all the way through the pre-release process without getting caught? What might we now do to this pre-release process so that such a thing doesn't fail to get caught again? Discuss this with the team – entirely without apology.

1

It doesn't hurt to say, "sorry, my bad" but that won't prevent it from happening again.

Take this as a learning opportunity and if anything use this as an to introduce the Org to a Post-Incident Review process. That should in a non-judgemental process, identify the procedural root causes that need improvement. See how Google handles their issues for example.

I should add that if you can achieve that, then you are no longer known as the person who introduced a bug to Production, rather the person who introduced a process to systemically eliminate bugs from getting into Production!

-3

On the presumption that the software defect in production was not due to intentional obscuration of the defective code, an apology email or message about the defect is unnecessary unless root cause analysis finds only you at fault through the root cause of the issue.

Ultimately, your team's review process and the testing you did was part of the process to try and avoid defects in the future, and that is the aspect being reviewed because it failed to catch a bug in production before it went live, and led to the team discovering it after the fact, and rolling back to a previous version.

I'm not sure of the nature of the Root Cause Analysis meetings you're in, whether your team is accepting blame for the failing of other team's downstream products as a result of your change, or if they're trying to figure out if the issue was upstream of your team and something your team needs to be aware of.

It sounds like it might be the former, given this:

I feel embarrassed for not recognising the issue in my code well in time and causing a lot of hassle for my team. However, fixing the issue and our process around deployments to ensure this does not happen again, has been a great learning experience for me as an engineer.

But if it was the latter, you're effectively doing the job of ensuring that their code is better, or that your team has a better understanding of the upstream code that your team is using - which is a defect that was being investigated to avoid other teams downstream of them from having a similar defect.

This all said, in either case, the question I have is: "How long after the issue was discovered that the cause of the defect was discovered and fixed?" It sounds like it's been about a week, or potentially less than that - it's only just Wednesday here, which means, if it was deployed on Friday, your team learned of the issue on Monday, and solved it by Wednesday.

That's pretty quick turnaround, especially given meetings, for tracking and resolving the issue.

Let me introduce you to the concept of Code Golf.

The concept of Code Golf in the questions linked to above is to, given a problem space, solve it in the least number of bytes. Given the language used in some of these cases, their code is intentionally more vague specifically to save bytes of filesize information.

One potential use of something like that is to obscure the goal of the software, and in doing so, would allow you to get something like these problem space solutions into areas they shouldn't necessarily be.

If your initial solution that led to the defect in your code was due to intentionally making it hard to catch the issue, and making them spend more time isolating what specifically went wrong, you very much probably should apologize to your team for making it into an almost "Gotcha" territory defect. Given how you've phrased it as a mistake that you yourself missed, that doesn't seem to be the case - and the quick turnaround on catching it means that the issue was easier to spot than some more difficult bugs.

It didn't get caught on the spot, or before it made it to production, but catching the initial reason for the defect was relatively easy to do after the team was alerted to it, despite the time pressure to find it. The resulting meetings afterwards were related to how to avoid causing that defect, but the issue was about the defect itself, not on how the defect managed to stick around for multiple releases due to being a hard to find defect in a messy codebase, nor one that intentionally meant to mislead others to approve it.

TL:DR; You'd apologize if you had something like a non-standard use of GOTO to achieve obscure ways of bypassing Dev and QA testing environments intentionally for an issue to land in production, but if that's not the case, you should be fine - everyone makes mistakes that make it to production before being found at least once.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .