43

I joined a hot startup as a developer two weeks ago, and one of my first tasks was to review the work of another developer, test it, and deploy it. This is my first job as a developer, coming from a mechanical engineering background.

The code itself wasn't that difficult. I reviewed it and tested it. It worked fine, and it's been working fine for the past week. I even watched it for a few days on our production server and it was succeeding 100% of the time, so I figured it was good to go and moved on to my next task.

Just today we spotted a bug on our production server where a small number of requests have started failing (way less than 1%). We traced it back to having only started yesterday.

We root caused it as being caused by the code I tested and deployed. And I'm pretty much being blamed for it.

The bug itself only happens extremely infrequently. The code base is legacy and the developer assumed that we could get our data from a specific object, but it turns out in an extremely rare case this object isn't populated (and believe me, there are thousands of lines of legacy code that goes through this object and it’s implied to always be populated by its usage and naming). There aren't any comments anywhere mentioning this can happen and the code itself has tons of technical debt.

I was under a tight deadline to complete this task too, and I feel that even if I had more time I wouldn't have found it either.

What should I do? Am I really at fault here? Did I screw up? What am I supposed to do in the future, so this doesn’t happen again?

17
  • 88
    By "being blamed" you mean? How did they communicate this to you?... what have you explained/justified so far?
    – DarkCygnus
    Commented Sep 7, 2022 at 0:25
  • 53
    What is a "hot startup"? Commented Sep 7, 2022 at 7:34
  • 43
    What repercussions does this blaming have? Is your job on the line? Mistakes happen. Even if the blame is justified it usually goes like "Yeah I really fucked up" and then you move on.
    – Ivo
    Commented Sep 7, 2022 at 8:17
  • 57
    @GregoryCurrie judging from the rest of the question: a burning mess. Commented Sep 7, 2022 at 9:00
  • 74
    Hot startup? Legacy code? Only one of these statements can be true at any given time.
    – MonkeyZeus
    Commented Sep 7, 2022 at 13:55

14 Answers 14

116

We root caused it as being caused by the code I tested and deployed

No, you didn't get to the root causes. Even if you weren't using any more comprehensive techniques, a 5 whys analysis would keep going. You found a part in the system that has a defect. Why did it take days to start failing? Why didn't the testing include the scenarios where it failed? What aspects of the process allowed the defect to make it into the codebase and the production environment?

I'm pretty much being blamed for it.

Are you being blamed for it? It's not clear what your colleagues, lead, and/or manager are saying. Without knowing more, it's hard to say if it's about blame - assigning you the responsibility for the defect - or if it's about helping you improve.

It's not helpful to blame people for failures. Failures are an opportunity to make improvements. This is why we have root cause analysis, blameless postmortems, and other forms of continuous improvement. There are opportunities to help people make better, more well-informed decisions and be more effective in their daily work.

What should I do? Am I really at fault here? Did I screw up? What am I supposed to do in the future so this doesn't happen again?

These are questions that only your lead or manager can answer.

Since people and human error are never root causes, I would suggest that even if you made a mistake and missed something, something in the process allowed you to make that mistake and then let that mistake flow all the way into production.

If you were doing true root cause analysis, the team would be getting to exactly what happened and taking steps to prevent it. There could be product changes to improve the code and pay down aspects of the technical debt to improve the understanding of the system. There could be process changes to make sure that more people, especially people with more knowledge of the system, are involved in the development process and teaching new hires. Maybe there are other solutions that will become apparent once you get to the true root cause(s).

6
  • 30
    "This is why we have root cause analysis, blameless postmortems, and other forms of continuous improvement" - +1 just for this, and I would +1 once more for the answer itself.
    – Bleh
    Commented Sep 7, 2022 at 17:52
  • "people and human error are never root causes" -- what about a case of intentional sabotage? Assuming that all reasonable mechanisms are in place to deter, detect, and mitigate, and it's a case of someone deliberately misusing the powers they had to be given to do their job, and as a result the sabotage inevitably causes some damage...isn't the root cause of that damage the person who chose to commit it? Sometimes, it really is a wrongdoer's fault, and asking "why" more than once can amount to victim-blaming.
    – nanoman
    Commented Sep 10, 2022 at 3:47
  • 1
    @nanoman Why did the person commit sabotage? They had some underlying motivation to do so. Commented Sep 10, 2022 at 9:48
  • You described root cause analysis as "getting to exactly what happened and taking steps to prevent it". The motivation of a previously trusted attacker is often unknowable (especially because anyone with an obvious motivation might have already been filtered out via "deter, detect, and mitigate"), or even if one can guess the attacker's motivation, it often indicates something that needs to be fixed within themselves (irrational hatred, impulsive greed, etc.) and not in the entity they attacked (who is doing the root cause analysis). ...
    – nanoman
    Commented Sep 10, 2022 at 21:07
  • 1
    @nanoman The most common motivators for malicious insider threats include dissatisfaction with some aspects of the job, money, or beliefs (political, social, etc.). There are process solutions for mitigating all of these. Human actions are only causal factors. Commented Sep 11, 2022 at 9:41
40

I would really like to hear how exactly they are blaming you. You don't make it very clear in your question and it's possible the situation is not as harsh as it seems from this brief explanation.

But assuming the premise of the question is true, this would be a fundamental failure on their part. No one should be blamed for the work they do in the first few months on a new code base, let alone as a brand new developer. I'd expect all your work to be meticulously reviewed in the beginning because it's 100% expected you will make mistakes. Whoever put you on this review task should already have gone through other steps to verify that code change, and the developer that actually wrote the code would take the brunt of the responsibility (but not blame, because blame culture is toxic)

As to how to improve your technical skills, I think there are better stackexchanges for that, this one should stay focused on workplace situations.

13
  • 9
    Its a legacy code base which was purchased by the startup from another company. We had a meeting and I was blamed for it in the meeting, told to be more careful in the future. But the bug just started happening now. When I tested it (and its verified from the logs) it wasnt there. The code is really bad, and some weird condition happens sometimes that causes the request to fail very very very infrequently. It would pretty much be impossible to reproduce without the exact sequence of events that happened in production yesterday. No idea how I'd even go about testing for this failure before.
    – Daniel
    Commented Sep 7, 2022 at 1:25
  • 11
    Code base turns into legacy at the exact moment a new developer inherits it. In a startup this can happen in weeks, and there's a good chance it was originally developed as a quick hack before seed funding runs out.
    – ojs
    Commented Sep 7, 2022 at 9:37
  • 16
    @SebastiaanvandenBroek - Startups often have a terrible legacy codebase because the CTO hashed together an MVP on their own without really knowing what they were doing before realising they need a team to help them... That team then has to deal with that. Whether that's the case here or not I couldn't say, but it's something I've seen time and time again as a consultant Commented Sep 7, 2022 at 10:59
  • 6
    @Benjamin - I don't buy that definition... It's very possible to have legacy code that does have automated tests and have new code that doesn't... Although lacking tests is a common feature of legacy codebases Commented Sep 7, 2022 at 12:01
  • 6
    Your sentence starting "No one should be blamed for the work they do" needed to have a period ending there. The only time someone should be blamed is if there is a consistent pattern demonstrating a lack of desire to do a good job (essentially, there is malice). ANY human endeavor that depends on vigilance is doomed to fail, and usually spectacularly. Humans cannot detect all bugs, catch all mistakes and sources of error, and so on. Especially in environments where there is pressure for features above all else. Failures must be treated as systemic FIRST.
    – CodeSeeker
    Commented Sep 7, 2022 at 17:23
37

No one person is responsible for bugs


People write buggy code because writing code is hard, that's just a fact of the industry.

As such, any functional team of developers will have multiple layers of protection in place to prevent bugs from getting to production, these could include:

  1. Pair programming
  2. Unit tests
  3. Integration tests
  4. Peer review
  5. Canary deployments

As such, for a bug to have reached production, all of these layers have to have failed. If these layers don't exist, then the team as a collective unit has failed to do its job effectively.

The long and short of it is that a bug reaching production is a collective responsibility of the team, not of you alone.


Now, how you should deal with this depends largely on how much you want to work at this start-up. If you're not that bothered, I'd look for a job with a less toxic development environment. If you do really want to work there, then its time to start having conversations with your team and CTO around how software craftsmanship keeps development velocity high, and defect rates low in a sustainable fashion. As well as how if you just pile tech debt on tech debt you'll slow to a crawl.

At that point, it becomes a no brainer to implement these systems.

10
  • 5
    Good answer. Bugs are a team effort. The first order of business is to figure out if the process can be improved to reduce bugs. Commented Sep 7, 2022 at 12:04
  • 2
    I love this answer. The only addendum I'd make to it is that the "how much do you want" part is important because if a team does not already realize and embrace all of the above, then they are both amateurs at writing software and not nice people to work with. So you'd need to "want it" a great deal in order to decide to put up with those things. Remember that all companies are desperate to hire people, especially software companies. Other amazing jobs are plentiful. Commented Sep 7, 2022 at 14:58
  • I would absolutely argue against this point in some circumstances. I have worked with 'hot shot' developers who 'need' to push their code 'now', and it's bug-free and absolutely works. Except, it doesn't, and their hotshotted deployment DID break everything. Following the established protocols? no one person is responsible. Deploying from your desktop? Yeah, YOU ARE RESPONSIBLE. This is a smaller-shop thing though.
    – Bryan B
    Commented Sep 7, 2022 at 15:58
  • 4
    It's not so much "people write buggy code because writing code is hard" but "you get the impression that people write lots of buggy code, because unlike most professions, your mistakes will be found out". How often does someone order a cocktail, it's not done correctly, and nobody notices? If I write a make_cocktail() method, and there is a bug in it, it will be found eventually, even if it is on average better than the human cocktail maker.
    – gnasher729
    Commented Sep 7, 2022 at 17:45
  • 2
    @BryanBoettcher - You shouldn't be able to push your code to production if the correct protocols are in place. It should have to go through a PR and the CI pipelines. Commented Sep 8, 2022 at 8:48
12

Step 1. Accept the blame, yes.

Step 2. But stand your ground and ask for what you want.

In other words, do not promise that this won't happen again. Do not apologize and do not make promises you can't keep. Do not even imply that you won't do this again. This part is very important.

People make mistakes. You made a mistake. That's completely normal. You're just a person, and a brand new engineer at that. Do not defend yourself. Own the part that you could have done better (in hindsight). But do not promise that you won't make those kinds of mistakes in the future. Do not stop looking for other root causes. And if you have suggestions that would improve the reliability of the system, work on pushing them through. Be relentless about it.

Because if they're blaming you, as a way to avoid dealing with the underlying issue. That's not good. They can only do this so many times before the entire system blows up in their face.

Also, you will continue to make mistakes. Everyone does. Even if you remain at that company 5 more years, you will continue to make mistakes 5 years from now (and this will continue to happen whether your suggestions get ignored or get implemented).

If you're intrigued by that approach, you should read "When I Say No I Feel Guilty" by Manuel J. Smith. I know the title of that book sounds wonky, but do not judge that book by its title. It's an extremely insightful book. Just read its customer reviews on Amazon (I know that Amazon customer reviews are not always reliable, but in the case of this book, they are).

Also, I want you to stop putting your company on a pedestal. Theranos and Pets.com were "hot" startups also. And in your case, I do not care if you think you're going to be a paper millionaire in a couple of years, but if what you described is remotely accurate, your company's engineering department is absolutely dog shit.

And from your perspective, as a brand new software engineer, this is the most important part that you should be worrying about. Because if the company fails, or if you get fired, or if you get blamed for every little bug that's present in the legacy code, that's not good for you and that's not good for your growth as an engineer.

8

It sounds like a bad error culture. It might be your misconception, but you should never blame someone for an error, but work together with all people involved to improve and to try to make such errors less frequent or less severe in the future.

A bad error culture is a negative point for a job, similar to bad boss, bad payment, or bad coworkers—it can be a reason to switch jobs.

1
  • 1
    Blaming a new starter for missing a very subtle bug (<1%, after a week) in code review is bananas Commented Sep 10, 2022 at 2:24
7

Lots of other great answers about the 'political' side of this issue. I see one technical aspect that isn't addressed, though.

This statement is the crux of your technical issue: "the developer assumed that we could get our data from a specific object but it turns out in an extremely rare case this object isn't populated"

You never know for sure if an object is populated unless you test it. Even a simple "if (object == null) { throw error }" could have saved you from this issue. Always be skeptical of code that doesn't test for success/failure. As a 30+ year developer, this is the best advice I can give you here.

1
  • Very true. Always check that the database has the expected rows in the table, that a connection actually receives data, that passed parameters are not null or empty or filled with valid values, etc., Everything eventually fails, and you want to catch before it starts processing it. Then you can blame other people, instead of getting blamed. :-) Commented Sep 14, 2022 at 9:43
6

If I understand it correctly... you just joined a startup, and judging by the term "hot startup", I get the idea that everything is happening at once, at all hours of the day. All the good, and all the bad. Duties change for some people every moment of every day, even during the off hours. Higher ups are no doubt pulling 25 hour workdays, 8 days a week. Figuratively, of course. (Honestly, if it's possible to do that, don't share lest bIg CoRpOrAtIoNs make employees work like that!)

First, it seems you come from a background that's a bit different than software development. Combine this with it being your first job, and no doubt this can be intimidating.

One thing I've found is that some of the "simplest" code ends up being the most annoying to contend with when something breaks - whether it breaks immediately or sometime down the road when seemingly nothing has changed.

So first... who is "we"? Who did a root cause? You're vague and as some other answers here imply, it seems there's a lot more to it than you've said - did you really run a root cause analysis? Did you see what exactly caused the issue? Did you even confirm that it's something that you even looked at, or had reason to look at but skipped?

Secondly, the phrase "pretty much" is really, really dangerous, in my experience. "Pretty much" means an assumption. "Pretty much" simplifies something immensely. It can be really powerful and useful to create concise, simple language, but you can pretty much not use the phrase "pretty much" when asking things on this site. Detail is key. Yes, I used that term just now. It's pretty much a matter of knowing when it's useful - I'm not asking for information so there's nothing to elaborate on, and the comment wasn't intrinsic to your question.

That said, what exactly are they saying, or doing? Are they treating you differently because of a mistake 2 weeks in? Did you mis-represent your skills and they thought you were a 10 year veteran developer (with 50 years experience in a 2 year old technology, lol) or are they aware that you're still pretty green and came from a different background? Are they acting like you should have known this "old software" like the back of your hand? What were their ACTUAL expectations? On one hand, I read this, and I get the vibes that you deal with ADHD and RSD (Rejection Sensitivity Dysphoria), like I do. I develop a tool at my job that helps improve the workflow of my actual job, and my team uses it. When something goes wrong, even after I test it, I feel this overwhelming feeling of crushing blame, that my co-workers are judging me, and I let them down. I feel like I'm to blame for any mess up (even if it's Chrome that crashed on an underpowered virtual workstation and not my code that was at fault), and suddenly I'm going to be sent packing, with a small box filled with my stuff on the desk. Granted, I'm not officially a developer, but the feeling is very real. Yet I remember how much praise I get for the tools I create, realizing... hey, mistakes happen. And we fix them.

Are your peers treating you (and the situation) like a child? Or, are they handling this like an adult, accepting that mistakes happen no matter how critical of a job it is?

Assuming your peers actually function like adults with a level head, I feel you should do the following:

  • Take responsibility. This doesn't mean you're at fault. This means you're going to be responsible for fixing the issue, whatever happened. Maybe it WAS an error on your part. Maybe any other veteran developer would've done the same stuff you did and wound up in the same spot as you. Who knows - but you now found a bug, and it's now time to fix it.

  • Are you at fault? Who knows. Look deeper, or don't look at all if you might get lost in self-thought. Discuss with your peers and see what happens.

  • What you're supposed to do... is treat it like a learning experience. You're not going to know everything. Even if you knew everything about the programming language and best practices, doesn't mean you know what has to happen HERE. Programming is a mess. Nothing is always perfect, and I found that just because perfectly written code is clean and might run great, doesn't mean that's what you'll be writing in production, or what you'll be debugging. It's not unusual to fix one thing that suddenly breaks somewhere else. It's a mess. Understand that this will probably happen again. Make sure your peers realize that this might happen again with ANYTHING. You might check everything that can be reasonably expected to happen (and everything that shouldn't happen but might), but the moment you deploy your code, something breaks anyway. All you can do is find the issue, fix it, move on. If the company your at actually has good people, you shouldn't be in any trouble.

  • The ONLY exception I might make is that in rare cases you're in a job that accuracy is key, even from the beginning. I've been in two such spots, but there was still a period that you could make mistakes if you were new. If you had prior experience, they were less lenient, but still somewhat accommodating - so while I could see things being worse if you're in this kind of company, I'd be inclined to believe that this is the exception, not the rule. And again, make sure people aren't acting like adult children.

5

Start Working On A Solution

In a proper code deployment schema, you would have thoroughly tested every edge case you could conceive before requesting a code review from a co-worker, then pushed it into a Testing environment, Staging environment, and finally into Production. At any step between Production and your own coding, someone should have been able to notice this bug, but didn't.

Looking for a person to blame for this failure is a fruitless task, because every person involved in the deployment of this change is responsible.

Regardless, you may still be blamed for this bug by your peers or even supervisor - because people are fallible and often look for a person to blame when things go wrong.

The correct response to finding a bug in production, regardless of who is to blame for it, is to start working on a solution for the bug.

You do not have to 'admit' to being the cause of this bug, just be honest about what you coded and how you coded it (as you've already described to us), and work on fixing it.

A good team lead will be grateful for any employee who works on a solution to the problem - so don't get too tangled up in figuring out who's to blame, just be honest about what happened and look for a way to fix the problem.

3
  • If the OP is being blamed in this situation, it doesn't sound like they have a good team lead... so they have to deal with that ahead of fixing the more technical aspects Commented Sep 10, 2022 at 3:04
  • @JiříBaum It's still a good idea to focus on finding a solution to the problem - even a bad supervisor appreciates results, and finding a solution and providing documentation takes some of the pressure off of OP while also providing proof against OP being negligent in their coding.
    – Zibbobz
    Commented Sep 12, 2022 at 14:12
  • Leading change as junior-most in a possibly toxic environment may or may not be fun, and certainly the bulk of the work will be non-technical Commented Sep 13, 2022 at 2:10
5

We root caused it as being caused by the code I tested and deployed and I'm pretty much being blamed for it.

What should I do? Am I really at fault here? Did I screw up? What am I supposed to do in the future so this doesnt happen again?

Since you didn't write the code, obviously you aren't solely at fault here.

You should take this as a learning exercise. Talk to your boss and/or team leader. Ask what you should have done differently. Perhaps this code should have been integration tested on a server deeply enough such that the unpopulated object would be exposed. Perhaps the codebase is such that each module must be coded extremely defensively, and not assume that any other object does what is expected. Perhaps you should have tested more. Perhaps you should have asked for help. It's not clear what would have made a difference - perhaps nothing.

Either way, don't be defensive about this. Mistakes happen. This is your first development job and you've only been there two weeks. And reviewing and testing code is completely different from developing it. This is all new to you.

Make sure to write a bug report about the failing object this isn't always populated and is the real source of the problem.

Learn and move on.

4

Anyone can blame you, doesn’t mean it’s your fault. If there was knowledge that the problem could happen but it wasn’t documented, someone should have documented it and it was their fault. If there was no knowledge then it was more on the primary developer to avoid this and less on you. And shit happens.

1
  • Oh, it's clearly NOT OP's fault. The one thing OP should avoid here is promising it will never happen again. Because obviously, it WILL happen again, and when it happens again, it WON'T be OP's fault. Again. Commented Sep 8, 2022 at 21:24
4

It doesn't seem very reasonable to entrust the testing of a legacy codebase to someone (a) with a mechanical engineering background and no experience as a developer, (b) who joined two weeks ago, and (c) is working under a tight deadline.

Even if you are very competent, you wouldn't expect to be holding full responsibility - you'd expect someone to be checking your work, not you checking someone else's work.

The only possibility of you being to "blame" is if you induced them to allow you to take responsibility against their better judgment (and have already dropped the ball), or if you didn't raise an objection sooner to something obviously beyond your (zero) accrued familiarity with the codebase.

3

Thomas Owens' answer is solid, but I think you might need to step back before you can appreciate it.

When you are accused of putting a bug into production (or causing any problem really) the first thing you should figure out is: did I cause the issue or contribute to its occurrence in some way? It's natural to jump to your own defense, but it's bad teamwork. Own your shit. If you really don't think it's your fault, prove it.

I have lost count of the number of times that I was sure it wasn't my fault, but when I went to prove it, it was actually my fault. Own your mistakes and you will earn trust. If it's not your error, try to help solve the issue and then worry about demonstrating to people why it wasn't your fault. Often, that last step won't be necessary.

1

What am I supposed to do in the future so this doesn't happen again ?

In a well-established company, usually, there are formal steps to execute a test plan as follows:

  1. The first step is to have a test plan or deployment plan written by a tester.

  2. The second step is to have this plan formally reviewed and approved by senior engineers or SMEs (Subject Matter Experts) on the domain.

  3. Finally, the tester can now execute the plan to verify that the new code works.

Since yours is a startup, there may not be any formal procedure or process in place. So, a good strategy is to come up with some brief notes on your own regarding your test cases, and then ask the team lead or senior engineers to provide extra inputs on the rare scenarios that you may not be aware of. Their inputs will always be very valuable. As time goes by, your experience will improve significantly.


Don't worry too much about the topic of "fault" and "blame" at this point.

Since you are a new developer and this is your first job, a reasonable manager knows that you need some time to grow and develop your skillsets. So, a good manager won't blame you for the bug in a negative way. Instead, the manager, team lead, and senior engineers should give you some good guidances if you know when and how to ask for their help in a productive way.

All developers and testers (good, bad, senior, entry level) would accidentally have bugs in their work at some points during their careers. There is no exception. (Of course, a super talented worker would have a lot less bugs than a careless worker :-)

-1

This happens all the time. Even with a good test suite, bugs are inevitable. Even in space travel there were a number of them.

So unless your company invests in mathematical proof of all your applications, then bugs will pop up from time to time.

As others asked - what does it mean that you were blamed? You need to be very specific to get an opinion.

In general it was your code so you didn't catch it. On the other hand, there were no tests to cover this edge case so it's unreasonable for others to believe that you had a big chance to catch it. So there should be no blame in this case. And if there is, they don't understand the software development process and they don't invest in writing enough tests.

I wrote an answer because another answer suggested documentation for edge cases. It is impossible to document all edge cases and it is impossible for one to remember all of them. That's why we write tests.

P.S. also sounds like you perform no code reviews which is another point where bugs can be found and responsibility shared between the team.

4
  • @JoeStrazzere, ok, so you must admit you didn't catch it. The original author also didn't. Now how feasible it was to catch it is another story. Also whether issue is big or small is another story. But fact is that these changes caused the issue. Trying to deny the facts is not giving you anything. You say that you didn't catch it and also that it was unfeasible to catch without a proper test. I hope you added a test now? Commented Sep 8, 2022 at 12:19
  • 1
    Unrelated, but mathematical proofs aren't really all that useful; you can prove that the code matches the proof, but you can't prove that either of them matches desired and/or desirable behaviour. Modern code development techniques do about as well, at lower cost. Commented Sep 10, 2022 at 2:18
  • @JiříBaum, show me your bugless code ;) Commented Sep 10, 2022 at 9:50
  • I mean, you get bugs either way; effectively the difference is whether you write your unit tests in C or Z Commented Sep 10, 2022 at 11:24

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .