How to navigate an embarrassing mistake I created and recover?

Question

asking for a friend

As part of a small project I wrote some code some months ago.

The code passed my tests, code review, UAT by business analysts and everything was fine. When the time came for me to request that it be loaded to production I had a gut feeling something wasn't right. I double checked the code and found a very serious logical error.

The problem was that between the time it was loaded to production and the time it went live (can be anywhere from a couple of days to a couple of weeks) the "fix" was going to deactivate another fix by someone else that was crucial to the business.

Embarresingly I reported it to management that I made a huge mistake.

The rest of the team has now also become aware since we're having meetings, sitting together assessing impact and discussing it openly. Senior management are also aware and want complete reports of what happened.

My manager is trying to help me remain calm telling me he is glad I spotted this before it went to production since we now have options. All options involve bringing our local office into the spotlight before the global corporation and having to explain how such a simple mistake was made in the first place. (This "simple mistake" isn't so simple or obvious but once it is pointed out it becomes very obvious, but it is difficult to pick up).

I've never made such a mistake in the workplace before and am extremely embarrassed about it and feel my "brand" has been tarnished. I am not sure if I should be apologetic, or just focus on finding a solution or just hand in my resignation now and move on. I feel terrible that I have wasted so much of the companies time on this already.

tl;dr What is the best way to navigate this situation when I am personally responsible for a huge mistake in the workplace that is pure human error?

EDIT:

As it turns out the whole thing was a false alarm - I messed up the boolean logic table (made a typo and didnt double check) when studying the logic which in turn gave the false result that the massive error would have happened. Since this code was written (by me of all people) around 6+ months ago I must have completely forgotten what I was trying to do but as it turns out it was correct the first time. I think this is worse, I panicked and wasted half a day of my time and the managers time for nothing.

Answer 1

When you panic over a potential bug, what you're really doing is pointing out a risk that the analysis failed to identify and the testers failed to test. That is a failure, even if the code ultimately worked.

People make mistakes, and companies have processes to allow for that fact. That's why there was a code review and UAT (and presumably some other testing).

This situation is a red flag to the manager, not that you are a poor programmer, but that the processes weren't tight enough. Did the reviewer just skim the code, only checking coding style? Did the testing only cover a few of the possible situations? Was the risk of a complex area underestimated? For a mistake to reach production, every person in the chain must also done something wrong.

In the case of a false alarm, the question becomes "why did the process not reassure you that everything was correct?". Are you working on something so complex that no one else understands it? Are you under pressure to finish before you've written all the unit tests you would have liked?

Organisations that penalise people for making mistakes find that future mistakes are fixed in secret, hidden, or blamed on other people. Safety critical industries like aviation learned a long time ago that they need to encourage people to own up to mistakes, so they can improve their process.

Similarly, don't beat yourself up because you're not perfect. Thinking that you can avoid mistakes is the real mistake. Instead assume that at some point you will forget about the obscure feature and look for a way to protect it from future-you on an off-day (or a new co-worker). Maybe more unit tests, input checking, error handling or even just a few more comments.

Answer 2

The difference between you and someone less capable is the gut-feel that something was wrong. This gut feel is borne by experience, and forged by previous mistakes.

The concern needs to shift from blame, onto how this situation can be avoided in the future. The best person to identify the process defect that allowed this issue to pass testing and review by multiple levels of the business is you.

You need to get rid of the mindset that it a pure human error. There is no such thing. There is human error coupled with defective processes, or multiple humans making errors, but if it's important, it can't be down to a single person making a single error.

Don't compound the businesses pain my jumping ship when it's time to knuckle down and improve things. If they ask for your resignation, you can decide what to do then, but until then, you have to keep working hard.

Answer 3

One thing you should consider:

You have set a shining example of how to properly deal with an issue.

Yes there is pain in sorting it but leaving it sounds like it would have been SO much worse.

Someone in management needs to stand up and say, in front of all, that they are pleased to be sorting this prior to the production phase and get all to focus on resolution.

The other question is how can you test for that type of error? What is needed and what should be added to the testing routines?