7

asking for a friend

As part of a small project I wrote some code some months ago.

The code passed my tests, code review, UAT by business analysts and everything was fine. When the time came for me to request that it be loaded to production I had a gut feeling something wasn't right. I double checked the code and found a very serious logical error.

The problem was that between the time it was loaded to production and the time it went live (can be anywhere from a couple of days to a couple of weeks) the "fix" was going to deactivate another fix by someone else that was crucial to the business.

Embarresingly I reported it to management that I made a huge mistake.

The rest of the team has now also become aware since we're having meetings, sitting together assessing impact and discussing it openly. Senior management are also aware and want complete reports of what happened.

My manager is trying to help me remain calm telling me he is glad I spotted this before it went to production since we now have options. All options involve bringing our local office into the spotlight before the global corporation and having to explain how such a simple mistake was made in the first place. (This "simple mistake" isn't so simple or obvious but once it is pointed out it becomes very obvious, but it is difficult to pick up).

I've never made such a mistake in the workplace before and am extremely embarrassed about it and feel my "brand" has been tarnished. I am not sure if I should be apologetic, or just focus on finding a solution or just hand in my resignation now and move on. I feel terrible that I have wasted so much of the companies time on this already.

tl;dr What is the best way to navigate this situation when I am personally responsible for a huge mistake in the workplace that is pure human error?

EDIT:

As it turns out the whole thing was a false alarm - I messed up the boolean logic table (made a typo and didnt double check) when studying the logic which in turn gave the false result that the massive error would have happened. Since this code was written (by me of all people) around 6+ months ago I must have completely forgotten what I was trying to do but as it turns out it was correct the first time. I think this is worse, I panicked and wasted half a day of my time and the managers time for nothing.

1
  • 6
    "I've never knowingly made such a mistake in the workplace before" - fixed that for you. We've all made mistakes, most are not caught by a gut-feel but by an angry customer. If anyone is giving you a hard time then they're clearly happier to cover up their own examples, which is a bad sign. Sep 2, 2019 at 5:41

3 Answers 3

6

When you panic over a potential bug, what you're really doing is pointing out a risk that the analysis failed to identify and the testers failed to test. That is a failure, even if the code ultimately worked.

People make mistakes, and companies have processes to allow for that fact. That's why there was a code review and UAT (and presumably some other testing).

This situation is a red flag to the manager, not that you are a poor programmer, but that the processes weren't tight enough. Did the reviewer just skim the code, only checking coding style? Did the testing only cover a few of the possible situations? Was the risk of a complex area underestimated? For a mistake to reach production, every person in the chain must also done something wrong.

In the case of a false alarm, the question becomes "why did the process not reassure you that everything was correct?". Are you working on something so complex that no one else understands it? Are you under pressure to finish before you've written all the unit tests you would have liked?

Organisations that penalise people for making mistakes find that future mistakes are fixed in secret, hidden, or blamed on other people. Safety critical industries like aviation learned a long time ago that they need to encourage people to own up to mistakes, so they can improve their process.

Similarly, don't beat yourself up because you're not perfect. Thinking that you can avoid mistakes is the real mistake. Instead assume that at some point you will forget about the obscure feature and look for a way to protect it from future-you on an off-day (or a new co-worker). Maybe more unit tests, input checking, error handling or even just a few more comments.

2
  • 1
    Yes to all of the above. But what I can't shake off is how I made the mistake in the first place, I guess I'm just angry at myself and having my ego bruised like this isn't the nicest feeling. I also have a fear that I might be seen as a little "dumb" by my peers.
    – solarflare
    Sep 2, 2019 at 22:39
  • @solarflare In that case I can only recommend working on yourself. Maybe your work offers training or counseling? This might be a good one to invest some time in.
    – Borgh
    Sep 3, 2019 at 8:48
20

The difference between you and someone less capable is the gut-feel that something was wrong. This gut feel is borne by experience, and forged by previous mistakes.

The concern needs to shift from blame, onto how this situation can be avoided in the future. The best person to identify the process defect that allowed this issue to pass testing and review by multiple levels of the business is you.

You need to get rid of the mindset that it a pure human error. There is no such thing. There is human error coupled with defective processes, or multiple humans making errors, but if it's important, it can't be down to a single person making a single error.

Don't compound the businesses pain my jumping ship when it's time to knuckle down and improve things. If they ask for your resignation, you can decide what to do then, but until then, you have to keep working hard.

4
  • Thanks for the reply, I think its embarrassment more than anything else at the moment. And the business is extremely complex so mistakes are a regular thing, I've seen far worse mistakes (real facepalm mistakes) make it through and only get picked up AFTER a prod load. I'm mostly interested in what approach I should take (ie apologetic or focused on fixing it and treat it as "s**t happens" - which is what I've been doing)
    – solarflare
    Sep 2, 2019 at 2:35
  • 1
    This answer is spot on. Don’t be embarrassed that it happened, be happy that you caught it in time. I don’t know what kind of industry this is, but it’s possible the cost of fixing it now might pale in comparison to how much they could have lost. Listen to Gregory Currie, focus on the solution, not the problem. If you’re already invested in making things right, then you’re doing the right thing.
    – Lindsey D
    Sep 2, 2019 at 4:24
  • I updated the question - as it turns out my code was fine but I got jumpy when I didnt understand my own logic from 6 months ago. I wish the earth could just open up and swallow me right now.
    – solarflare
    Sep 2, 2019 at 5:28
  • forged by previous mistakes the most important part here. Making a mistake is fine, learning from it is not. We all mess up, learn from the issue and don't repeat it. You'll be fine. Sep 2, 2019 at 13:20
7

One thing you should consider:

You have set a shining example of how to properly deal with an issue.

Yes there is pain in sorting it but leaving it sounds like it would have been SO much worse.

Someone in management needs to stand up and say, in front of all, that they are pleased to be sorting this prior to the production phase and get all to focus on resolution.

The other question is how can you test for that type of error? What is needed and what should be added to the testing routines?

6
  • There was no error, I updated the question.
    – solarflare
    Sep 2, 2019 at 5:28
  • You guys should start a club or something. Sep 2, 2019 at 5:35
  • @GregoryCurrie we are in one, SO, and you are a “member” too...
    – Solar Mike
    Sep 2, 2019 at 5:38
  • Hey I'm having the worse day of my life here!
    – solarflare
    Sep 2, 2019 at 5:47
  • 1
    @solarflare no, there are others worse off, it just feels bad ‘cos you are the centre of attention. I used to work in a workshop and above each bench was a nail. There was an award for c*ck-up of the week, awarded to the mechanic who made the most expensive mistake in the past week. Probably not allowed now not “PC”... So as Douglas Adams said “Don’t Panic”... it will calm down.
    – Solar Mike
    Sep 2, 2019 at 5:52

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .