71

I am a software team lead. One of my guys built an update for android, but left the environment on "staging" and now everybody is freaking out: the marketing guys, PM, product owner.

So far I have taken the "blame", and it is my fault really for not checking before uploading... The app had to be unpublished, and the new fixed version appears that it will take some days for review...

Shall I mention that my colleague built the app in the wrong environment?

Or just take the blame and hopefully expect for the best (and of course check from now on)?

  • 4
    Take blame from whom? – Tymoteusz Paul Sep 17 at 7:32
  • 22
    How do you plan to "take the blame" though, send a mass email, or do you expect disciplinary action? And It is not so clear from your question, but you hint at that this is indeed your fault as you were supposed to check it, is that the case that you were supposed to verify the env? – Tymoteusz Paul Sep 17 at 7:37
  • 49
    Shall I mention that my colleague built the app in the wrong environment? Or just take the blame and hopefully expect for the best. - If you're the team lead then you take the responsibility for the work of the team. – joeqwerty Sep 17 at 11:45
  • 13
    Don't you have a build server? – Thorbjørn Ravn Andersen Sep 17 at 16:57
  • 11
    @joeqwerty Please do not use backquotes to highlight quoted text in comments. This syntax should be reserved for code or data, not normal text. Abusing code markdown has ugly results, causes problems for parsing tools such as screen readers for the visually impaired, and is easily avoided by using italics and quotation marks instead. – Lilienthal Sep 18 at 15:11

11 Answers 11

297

You are the team lead and so you are the shield protecting your developers from above. Pointing fingers might be good for you in the short term, but it will hurt team morale in the long term and come around to bite you. Blame culture is toxic and good programmers rarely stay around in one.

While the reason it happened might have been one specific developer, your whole team (and you) are at fault for letting it get that far. Your process for deployment is obviously lacking, which is specifically your job to fix.

My response to the complaints from above would be to promise to implement measures to prevent it from ever happening again while accepting blame for it having happened. Subsequently, I would implement code reviews, rework the deployment process and add automated testing that checks a build for correct flags before it gets pushed.

Afterwards, if you go that route, you can present those measures to those above, which can remove the black mark you have on your record. After all, mistakes happen, but you can show that you did something about it.

| improve this answer | |
  • 75
    "... while accepting blame for it having happened" - Perhaps it's better to speak of "accepting responsibility" instead of "accepting blame"? – marcelm Sep 17 at 15:51
  • 38
    An anecdote to support this -- When I was a junior developer, I made a change and accidentally left a file handle open. That segment of code gets called a few times per hour, and we didn't find out until it was in production and after a few months a customer's machine got knocked offline due to open file handles. That's super bad. When we discovered what the error was, instead of playing the blame game (Who did this-Who reviewed-Who tested it?!), my boss calmly said "We need to get this fix out," supported us, kept morale high, and we had a fix in the field in 3 days (That's very fast for us). – Davy M Sep 17 at 18:12
  • 64
    And remember every production failure has at least two root causes. One is why it happened, and the other is why it made it to production. – David Browne - Microsoft Sep 17 at 19:23
  • 30
    @DavyM Interesting! I have another: Decades ago when I was a junior, something bad happened, typical dev content sent to production. My Lead's manager called the whole team for a meeting and demanded to know who did it, clearly looking for someone to blame. My Lead said that she was the one to blame. Asked again, she repeated that and said that if he wanted to fire someone, that would be her. From that point on, everyone knew that she had our collective backs. Morale was sky high for years until she moved to greener pastures. – OnoSendai Sep 18 at 4:04
  • 8
    OTOH, I've had managers (at the same company) pull the team together and say "I took a lot of heat for this. Let's get it fixed and move on." Other than identifying the problem and implementing the fix, that was the end of the discussion. I was very happy to stay in that department and enjoyed working for that manager. – FreeMan Sep 18 at 11:51
43

You are responsible for the actions of your team, and they are responsible to you.

I have seen this in dealing with clients where I had made a potentially costly mistake with their job. When meeting with the client, my director completely had my back and took ownership of what had been done (as well as arguing why it wasn't such a big problem and that part of the error was on the client's side). However, away from that meeting, he made it very clear to me what I had done wrong and what needed to change.

Your situation is slightly different in that it seems the blame is internal, not external, but the same principle holds - you are the team lead therefore you are responsible to those above. If you say "so-and-so messed it up", you are just acknowledging your failure in managing your team.

That is why management positions come with higher pay - because they are taking greater responsibility.

| improve this answer | |
  • 2
    I like this because there is enough blame to go around. e.g. Why was it allowed that staging could be pushed to production? Tight deadlines, lax controls, etc? – paulj Sep 18 at 15:02
  • 1
    This. When communicating to the broader world, the good things are credited to my team, the mistakes are my responsibility. Frankly, standing tall for your team builds credibility not only with them, but also with stakeholders that work with you. – GeekInOhio Sep 22 at 16:08
28

"Blame" is the wrong word to use here. I think you should use the word "responsibility".

"Blame" implies finger-pointing, etc. It fairly screams scream-and-shout. All in all, it's an unprofessional term to use, and it's an unprofessional thing to do. I encourage you to be professional* about this.

"Responsibility" is a more professional term. And since you're the team lead, you must take responsibility for this. Sure, someone on your team screwed up. When one person messes up, the whole team has messed up, and as the leader you take the hit. Don't worry, it's temporary - and hopefully it motivates you to improve your team's processes-and-procedures so that it doesn't happen again.


*What do I mean by "professional"?

First, you cannot "blame" anyone - on your team, on another team, or elsewhere in the company - no matter what happened. Even if everyone knows that Bob did it, you don't get up in a meeting and say "BOB DID IT!". When things go badly you have to go all in on calm-reasoned-and-professionally-impersonal - especially when things go badly! You need to say, "We had an incident". You have to say "We need to improve our process so that X, Y, and/or Z cannot happen again. The reputation of this team within the company is too good to allow it to be tarnished by minor oversights. How can we improve this?". When someone says "Well, we can fire Bob!" and the meeting threatens to blow up into a blaming match, you have to be the one who steps in and says something like, "This is not about personalities. Nobody on this team is perfect - and no matter how 'careful' we try to be, unless our process prevents us from doing X/Y/Z then it will happen again. Now, how can we rearchitect our process to ensure that things go smoothly in the future?". You have to be the one to redirect away from making Bob (name chosen because it's mine/not all Bobs are like me/features and options will affect final purchase price/see dealer for full details) the scapegoat and put the focus on process. Processes are repeatable (if designed properly) - people are not. You have to be the one to ask the team how to fix things - even if you think you already know How To Do It you need to ask the team - because, y'know, they might have some really great/fantastic/better ideas.

| improve this answer | |
  • "hopefully it motivates you". There is no hopefully. The view from above should be something like : X happened, and escaped. As team leader, I take responsibility. Therefore we have already done Y (or at least, are now doing Y) to ensure X cannot escape again. – Brian Drummond Sep 18 at 11:25
  • 1
    Blame and responsibility are pretty synonyms here. I don't know why people like to play these semantic games, and what they think it will achieve, but from my experience it just annoys people who are managed. – Davor Sep 18 at 13:15
  • 2
    @Davor Their implied meaning is very different, which is the point this answer wants to make. One is about identifying people who screwed up the other is about identifying where the process failed and how that can be prevented in future. Though I think this answer could be much improved by expanding on this some more to make it clearer and folding the footnote into the main answer. Right now the footnote is about twice as long as the actual answer. I'd also encourage you to use italics in addition to quotes when you have such long quoted exchanges and to use single quotes for 'blame' etc. – Lilienthal Sep 18 at 15:05
  • 2
    @Davor when being managed I'm much more annoyed if someone blames me than if someone makes it my responsibility. Semantics matter. Correcting someone in which words to choose isn't about picking the right nice sounding words, it's about changing the meaning of what they say. It's not "you cannot use those words", it's "you should not think the way your words imply you think". – Frank Hopkins Sep 18 at 20:58
11

So far I have taken the "blame", and it is my fault really for not checking before uploading... The app had to be unpublished, and the new fixed version appears that it will take some days for review...

But is it really though? If a team makes a costly mistake, why does your company let a single team make costly makes? Where is the safeguards? Where is the automated tests? In reality things are more complex and there isn't a single person who is at fault and there is very little value in pointing fingers.

But a broken app has been published and that has had a negative impact on the business so it is worth to make sure that it doesn't happen again. Instead assigning blame to a person, you should look at the process that let to this. And you might find things like this (not exactly, since I am making this up):

  • A single team member made a manual build of an app?
  • The team lead did not check the build version, but there was not detailed guideline in how to do it
  • Someone tested the app but didn't have clear instructions what to test
  • There was no automatic build server
  • There were no guidelines to do smoke tests
  • No higher managers were involved in the release decision
  • No experienced testers were around because it was Friday past 6

If you look at all the facts, you will probably notice that more than one thing let to this issue and instead of pointing fingers you should focus on fixing the process so that it doesn't happen again.

| improve this answer | |
7

If this is really about blame you need to find another employment.

However, I don't think it's about blame. As Bob says it's about responsibility. And it's about making sure similar things doesn't happen again.

You need to figure out WHY this could happened, and then change the process to avoid those mistakes. This should be done with the team. Note that you have to dig a bit to ask the correct questions.

Examples:

  • Q: Why did a developer build for the wrong environment?
  • A1: The developer was stressed
  • A2: The environment have to be set manually during build
  • A3: The environment is hard coded, and have to be set manually before building

All of the above answers could be true. But the answers just leads to more questions:

  • Q: Why was the developer stressed?
  • A1: The developer didn't have enough time to do a release
  • A2: Other tasks should be delivered as well
  • A3: The release wasn't ready in time, but the release wasn't postponed

And so on. Make sure you ask the hard questions, that challenges your processes. Above I've intentionally left out what I think are the biggest problems in this situation. Those questions are:

  • Q1: Why did a developer do a release build?
  • Q2: How could the environment be left on "staging"?
  • Q3: How come the mistake wasn't found after the build was done but before sending the app to Google?

Once you know what the actual problems are you can look for solutions:

  • S1: A build system should build for test, staging and production environments.
  • S2: It shouldn't be possible to forget to change an environment, that just leads to problems similar to this one. Ideally the build shouldn't be environment specific. But since you are working with Android this might not be possible. You might want to build for all environments, making it clearly visible what environment you use.
  • S3: Automatic and manual tests before release. Some way to get the build information in the application.

When you got all of this information communicate with the rest of the organization, that way you show that you and your team takes responsibility. Don't go into detail unless asked, but make sure they know you have made efforts to minimize the risk of similar things happening again. And make sure to implement the changes suggested.

| improve this answer | |
5

Shall I mention that my colleague built the app in the wrong environment?

No. There's no need for anyone to know the name of the specific person who made the mistake.

The team carries the fault. And as the leader, you are the public face of the team.

Or just take the blame and hopefully expect for the best (and of course check from now on)?

Mistakes happen. Blame is not the issue here.

As the leader, you need to speak with the individual and make sure they know what they did wrong, and the impact it has on other teams.

Then, you need to review your processes and determine how you can make sure this mistake never happens again.

| improve this answer | |
3

There is blame, and there is responsibility.

It is your team, and you are responsible. The buck stops with you.

That said, the BLAME goes to the one who made the mistake, but you're not going to blame him or throw him under the bus.

The professional way to approach this is to accept responsibility for your team, nothing about the workings of your team gets discussed outside your team.

You take the hits with the higher-ups, and say simply:

We found the cause of the problem, and have corrected it, on behalf of my team, I apologize.

And leave it at that.

Your team member made a mistake, have a long talk with him and explain what happened, and ask him how he will ensure that it does not happen again, and leave it be unless it happens again.

If there was no economic impact, and the other groups continue to harp on you, simple repeat:

The problem was caught in time, and corrected. Thank you for your concern.

The most important thing in a team is trust. As a team leader, if you threw one of your people to the wolves, not only would you lose HIS trust, but the trust on everyone in the team, as they'd know you wouldn't protect them.

Also, if I were in the position of any of the stake holders, I would lose respect for you as well. It looks very bad when a team leader blames one of his team for anything, and I'd assume you were weak or incompetent, or both.

Standing up, and taking RESPONSIBILITY for your actions, and the actions of your team is what a leader does. Assigning blame is not.

| improve this answer | |
  • 1
    awesome insights about leadership, thank you – manuelBetancurt Sep 18 at 17:14
3

If you've read Phoenix Project or Unicorn Project, you would know: in a good environment, no one gets "the blame". Environments with finger-pointing and shaming are not good environments.

It's not about "Who did this?!" It's about "What happened? Why did it happen? What can we do to prevent it from happening in the future?"

If you or someone on your team "gets the blame" or "is responsible" for what happened, you will all be more likely to hide your mistakes, rather than let each other know what happened.

When talking to those outside of the team, instead of "One of my team members caused X to happen. I talked to the person who did it and told them not to do that again." Instead, a more helpful approach is more like "X happened. We put policies in place to prevent X from happening again." If you still feel the need to phrase it as "So-and-so did X", instead of saying "I let this happen" or "My team member accidentally did this", maybe try "We accidentally did this". The problem lies in the process, not necessarily on an individual.

As for "blame" vs "responsibility/accountability", it is smearing some syntactic sugar on that wound. But as Mary Poppins said "A spoonful of sugar helps the medicine go down". It's slightly easier to swallow, and less finger-pointy. But it can still lead to the same exact issues.

However, if there are not serious negative consequences for the specific person who you're blaming, then I wouldn't worry about it. But don't throw any team-member under the bus for a single incident. If they get hurt because you said stuff, then you'll be less likely to find out when these things happen, and how they happen, and how to fix them.


Btw, for your specific case, you should look into automated pipelines which build the app for you in set environments. That's what we do for our applications. We have a pipeline that runs when we push to dev, which tests and deploys the dev version of the code, a pipeline that builds and deploys the staging version, and a pipeline that builds and deploys the production version. All three have to run by the same set of automated tests. Or if you can't do automated pipelines, then at the very least, the other environments should build to a file with 'staging' in the name.

| improve this answer | |
1

Your job is to shield the team, so you take the heat, but not the blame.

Explain to management that one of your developers made a mistake and that your team, including you, failed to find it and your procedures were insufficient to prevent it.

Explain what you are going to do to ensure such a mistake cannot happen a second time.

Case closed.

Under no circumstances reveal the name of the developer who made the mistake to management. If they press, switch from "someone on my team made a mistake" to "my team made a mistake".

| improve this answer | |
1

This is a quality problem. Quality problems are never individuals persons fault, but a fault of the team. If a single misclick in a GUI can result in a bad situation with direct consequences then the right solution is not to blame the person making the mistake, but put it in a checklist and establish a review principle for the critical step, where a second person reviews according to the checklist.

The answer to such problems is therefore for the lead: I am sorry that this slipped trough our procedures right now, we will include the problem in our future reviews.

| improve this answer | |
-1

Being a leader you are responsible for every error that you and your team makes and you help the team in every possible way, you can pass the blame if you are Boss. It all depends on your thought process.

| improve this answer | |
  • 3
    Welcome to The Workplace Sam Kruger! On this site we're looking for answers with some depth that explain why and how. Our goal is to build a library of knowledge for navigating the professional workplace. Please consider an edit to expand, and be sure to answer the full question. – Lilienthal Sep 18 at 15:06

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .