11

So I accidentally left a line of code in our application that caused a major accounting display error on the application and slowed down our clients' workflow for the entire day, forcing us to re-deploy the application rapidly today.

I've already owned up to making the error (It's not as if I could've denied it if I wanted - my name is there in the comments, as per our programming protocol) and everyone on the team is working to fix it (I can't do anything right now until we've finished preparing our code for re-deployment).

I'm not asking about how to handle fixing this problem (we are already doing that now). My concern is the aftermath - the fact that after this, the team I work with and my boss will have lost a lot of confidence in my coding ability.

How do I handle the blowback of making such a mistake? How do I minimize the damage to the team, to myself, and to my career?

  • 6
    If the only thing linking you to the code is a comment then your team has bigger problems. – Nigel Harper Mar 30 '15 at 14:17
  • 21
    Reminds me of a quote attributed to Thomas John Watson, Sr.... "Recently, I was asked if I was going to fire an employee who made a mistake that cost the company 600,000. No, I replied, I just spent 600,000 training him." Hopefully your management and team have a similar philosophy. Mistakes happen, they can be costly. Hopefully you'll be given an opportunity to show you've learned and you can be trusted in similar situations again. Remember this lesson when someone else makes a mistake that costs you dearly. – Kent A. Mar 30 '15 at 14:25
  • 3
    Given the quality of the answers here and the actual conte4nt of what he was asking, this is NOT a duplicate. The otehr was asking if you should apoligise with treats, this is asking how to move on from a mistake and recover, two totally different subjects and two totally differnt sets of answers. This shoudl be reopened. – HLGEM Mar 30 '15 at 16:46
  • 8
    I've never heard of a programmer who doesn't make mistakes. What is your team's testing procedure like? Maybe that's where the changes need to be made. Everyone will do something wrong or misunderstand something at some point. – Brandin Mar 30 '15 at 16:50
  • 4
    Voting to reopen. The marked duplicate is about compensation, whereas the answers here are mostly about preventing recurrence. – DJClayworth Mar 30 '15 at 16:54
18

The first thing that comes to my mind is "Do you ever check your own work"? If you ascribe the FUBAR to your coding ability, then it doesn't look like you learned anything from your experience.

When we code, we make mistakes in the coding - that's a given. Don't want to make mistakes coding? Don't code. The FUBAR raises the question "What do you guys do that resembles quality control in your neck of the wood?"

If you got away with changing code and not testing it before deploying it to production, then something is very wrong either with how you go about deploying code to production as a team or something is very wrong with the way you complied with the procedure for deployment to production. Which is it?

Owning up does not do anything for or against your credibility. Laying out explicitly what steps you are going to take to make sure that you don't repeat this episode - that's what's going to make the difference. Your team needs to have the confidence that you are not going to be careless - ever, about your code deployments to production. What are you doing to build that confidence? Because at this point, if I were either the team or the manager, my confidence in the reliability of your work is nuked.

And if I were your manager or your team lead, I'd be looking at some very tough moments with my own management right now. you might get away with a bad performance review while I could be fired because this episode occurred on my team and on my watch. You are not the only one who is getting damaged.

General advice: strange thing to say but I have been able to come out of some screwups with my credibility enhanced. The ingredients? Readily owning up to anything I did or didn't do. Full cooperation with anyone who is tasked with investigating. Doing what it takes to make the task of the investigators easy. Sharing thoughts in a coherent, straightforward, uncomplicating manner - mumbling, stuttering, hesitating are cumulative confidence killers. Not taking ownership of others' mistakes - what if you took ownership of someone else's mistake, you swear up and down that you'll never repeat that mistake - and the author of the original mistake does it again, and he is someone over whom you have neither control nor influence? Self-inflicted wounds are the worst.

| improve this answer | |
  • 2
    It was a minor change which I did test. The issue is that our code is a tangled mess of spaghetti code, and while I did test for the actual change I made, I didn't realize it was connected to another part of the application in a particular and unexpected way (Essentially, for the code-savvy, it was a null check for parameters, where I didn't realize that sometimes a null was expected for that part of the code). – Zibbobz Mar 30 '15 at 14:17
  • +1 for laying out steps. Basically write yourself up. Put yourself on a PIP. Cover your manager's tail as well. – Brian Mar 30 '15 at 14:18
  • 2
    What this answer describes is called cost of poor quality and every It organization should have ways to minimize it. If your code is not full of bugs all the time, there is no reason to give you a bad eval. Testing and especially regression testing in your case should find such issues. Other methods would be code review by peers, test scripts and test robots. It is your boss' job to introduce some of these measures to improve quality. – takacsmark Mar 30 '15 at 14:28
  • 1
    Your code change should have broken the other part of the application then and the breakage in that other part of the application should have been detected. It wasn't, either because there was no TDD in that other part of the application or because you never looked in that direction - I have no idea which is which. The question that comes to mind is why the code pass your test and then flunk when you deployed it to production? Why the difference in the outcome? – Vietnhi Phuvan Mar 30 '15 at 14:29
  • 1
    Would like to add that while it appears you've owned up to it, make sure it's known to the chain of command that you're responsible. Aka let it rain on yourself. Senior management are generally forgiving of a few mistakes here and there. Meanwhile the management you have to work with day-to-day are likely to regain respect for you when they realize they don't have to point fingers (because you already did, at yourself). Humility and self-sacrifice can go a long ways... – tekiegreg Mar 30 '15 at 21:02
10

First, don't panic. Stuff like this happens, and it probably won't damage your career more than temporarily. You've already done the first important thing, which is to take responsibility for your mistake.

The main focus from now on, for both you and your team, is to try to make sure that something like this doesn't happen again. Everybody makes mistakes, and the team should try to make sure that mistakes don't turn into catastrophes. It shouldn't have been possible for a coding mistake like this to get into released code.

Start by thinking about how this happened. Were you careless? Did you check your work? Did you execute whatever testing procedure your team has? Resolve not to do it again.

I would also consider if there is something the team could do to prevent this happening again - not just if you make a mistake, but one of your colleagues too. Could you add a testing step to the procedures? Or maybe a code review? It might be a good idea to go to your boss and suggest (without trying to avoid the blame) that one of those might prevent things like this happening again.

| improve this answer | |
  • Late Friday nights happen all the time. Mistakes are awesome in that they provide guidance how to improve things. – Vorac Jul 19 at 7:45
4

One thing I'd add to the other answers, and which was counterintuitive for me to learn, is not to apologize too much. You should clearly, publicly own up to your mistake once. After that, it makes sense to bring it up if you're doing a post-mortem on the issue or if you're discussing concrete steps to take going forward. Otherwise, let it be (but don't try to evade responsibility if someone else introduces the topic). If your team will be implementing new procedures or otherwise recovering, make sure you're an enthusiastic participant in the progress, but be positive about it.

This is how many people naturally handle things, so it may not be an issue for you. For my part, there have been times when I thought I'd screwed something up badly enough that I needed to apologize or take responsibility repeatedly, but my employers and colleagues were really more interested in moving on. If they're trying to forgive and forget, repeated apologies can make it harder. It may also cause them to perceive you as looking for reassurance, which ends up putting an additional burden on them.

| improve this answer | |
  • 1
    It can also reinforce negative perceptions you have of yourself, which can easily drag down your performance. Accept and admit your mistake, do your best to fix things, and move forward. No need to continually self flagellate. – Esoteric Screen Name Mar 31 '15 at 6:31
3

If your bosses see this as a reflection on your coding ability they are missing the point. Although you introduced the initial problem the team as a whole failed in allowing it to get into production. The focus should be on working out, as a team, why this happened and how to change your practises and processes to reduce the chance of it happening again.

The one thing missing from your story is any evidence of independent verification of your fix. Where I work no code is considered for release until it has had some level of testing or scrutiny by another member of the team. This isn't perfect but it increases the chance that the other party will know something you don't ("actually, the bar module passes in NULL and that's fine"), or at least think of a question you hadn't ("are you sure you've checked all the places foo is called from").

| improve this answer | |
3

Well - I'd say you've taken the first step (which many people omit) - admit to causing the problem and focus on solutions moving forward. As they say - "the first step is admitting you have a problem" - and you'd be surprised at how much effort people will go to to avoid admitting it.

Next steps...

The difference between human error and gross negligence

With a really bad mistake, there's a difference between human error and gross negligence. The difference is usually the difference in following the process you were given to the best of your abilities and understanding and still making the mistake (human error) or willfully/ignorantly NOT following a process when you should have known better. For example, if you were supposed to get a code review, and you didn't - it's gross negligence. If you did get the code review, and both yourself and the reviewer missed it - it's human error.

In cases of gross negligence, this really could have job/career impacting consequences. The action that you can take is to do the process you are told to do next time, but in essence you've broken faith with your employer and they don't necessarily have a reason to give you a second chance. This can be pay cuts, lack of bonus awards, or even termination for cause.

In the case of human error - the consequences are usually less severe - you may still get a bad review, or loose your bonus - but if it's an honest mistake, it's much more likely you'll get a warning. If you don't habitually make bad mistakes, then you'll dig your way out of the hole with good behavior and your reputation will heal.

First step - trace through the formal processes of your company and identify any steps you may have missed. Being able to say to your boss "I missed this step, I found it, did the best I could to recover and have these plans in mind to change my behavior so I won't miss it next time" - goes a long way in terms of rectifying the bad impression. And if you have a case where there is no part of the process that would have saved you - look for opportunities to propose a way to improve the process - either for your self or everyone. One way to look at human error type mistakes is "this could have happened to anyone".

Look for patterns

Your reputation will recover if this is your only serious mistake for a long time. Everyone commits a really bad error once in a great while. The people who usually suffer long term damage to their careers are those who repeatedly cause problems and show no significant improvement over time.

To avoid being one of those people - look for patterns that could have led to the problem. Did you have all the knowledge you needed to NOT make the mistake? Is there other preparation/error checking you could have done? Is there a work/life pattern that led to you being at less than your best (over tired, under-fed, stressed about home life, distracted by interruptions, sick, under the influence of anything not prescribed by a doctor, etc)? Can you eliminate any barriers to thinking and focusing efficiently?

Cut yourself slack on anything that you can be relatively sure is not likely to recur - for example, if you were tired because you had just spent all night staying up with your child in the emergency room of a hospital, you can be pretty sure that that won't happen every week or month. However, if you were really tired because you have a weekly volunteer commitment that keeps you up late - you may need to rethink the commitment and find a way to get to bed earlier.

Admit, Correct, Move on

Once you've looked at yourself and the patterns in your work and done what you can to prevent the error ... admit it, fix it, and move on. You'll get a certain amount of negative feedback for a while - try not to be defensive. Prove your worth by making conscious changes to your processes. And don't let it haunt you. Quite often, I see people still bringing up their mistakes long after the rest of the office has forgotten about it - once you are forgiven, let it go.

Do ask for feedback, however, 6 months and 1 year after the event, check in that your work quality has been better and that there's nothing else you should fix or pay attention to.

| improve this answer | |
  • There's a good reason why gross negligence exists as a term - there's also ordinary negligence. I think you're contrasting "human error" versus "gross negligence" while ignoring the middle field. E.g. you did get that code review, but that brought up "minor" issues that in hindsight caused the big problem. – MSalters Apr 1 '15 at 15:36
2

Manage yourself like a respected startup. After all, you are your brand. Look at Internet companies which have made mistakes, and see how the best of them have handled those mistakes. Then, you do the same.

E.g., the best companies and self-managing people begin an immediate process of;

  1. fully investigating the nature of the error,
  2. determining what changes in their process would prevent this kind of error from ever happening again,
  3. beginning to implement these process changes, and finally —
  4. reporting all the above transparently to stakeholders (i.e. your boss and colleagues)

If you do the above (and again, look for good examples of this done well) you'll inspire even more confidence than if this had never happened in the first place.

| improve this answer | |
1

You've done the first thing, which is to properly own up, and to take your part in the recovery process.

What you need to do next is demonstrate that you've learnt from the mistake. How you do this is a trickier proposition, but some approaches are:

  • Talk to your boss, apologising and stating that you'll take more care
  • Set a process for yourself to check these things - write it down
  • Look at some technical ways of ensuring your code is as correct as possible.

For option 3, maybe look at more automated testing?

| improve this answer | |
  • Find out if there is a regression.test suite. If not, suggest creating one. If there is one, fond out why it didn't catch your bug before the code shipped, and improve the tests or procedure as appropriate. – keshlam Mar 30 '15 at 14:28
  • @keshlam It's not a bug per-se...it's just a change that inhibits application functionality. It doesn't throw an error - it returns an incorrect value and assumes it's correct. It's...happy. :/ – Zibbobz Mar 30 '15 at 14:29
  • @Zibbobz that's exactly what a test suite is designed to find. You write a test, saying the value returned will be 6. You run the test, the value is 7 and the test fails. You fix the functionality, and most importantly it never leaves your PC, let alone gets to production – TrueDub Mar 30 '15 at 14:32
  • 1
    Wrong response is still a bug. It may be a bug elsewhere that your change exposed, but it's a bug. – keshlam Mar 30 '15 at 14:32
1

Mistakes do happen and pretending otherwise is just unrealistic

You already owned up, so that's good. The next question is "How do we turn this into a learning opportunity". Obviously you own an answer to "What will you do differently to make sure this never happens again" but there is more:

Now, this is not entirely your fault alone (although you should never say this out loud). Every once a while a whoppers get into the code, but apparently there isn't a good software process in place that prevents it from getting shipped.

You can use this to take it a step further: "Hey boss. I really screwed this up and now I'm thinking about how we can operate differently to make sure things like this don't get out". Do some research on good software release practices such as code reviews, check ins rules, testing cycles, automated testing, test coverage analysis, etc. Then come up with some specific suggestions that the team may be able to implement.

So in short, if you can turn this into a learning opportunity and at the end your department becomes better because of you initial mistake, you'll be fine.

| improve this answer | |

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .