Written warning from employer for following manager's instruction to deploy bugfix directly to production

Question

I'm wondering whether you wonderful people here could offer me some advice on how to deal with a written warning that I received today.

Background

I've been working for a software company for the last 8 years who deals with broadcast email. Clients use our service to upload their customer lists and send them promotional emails, track email responses etc. Some of these emails can be sent out to hundreds of thousands of people at a time.

There was a problem with one of the applications involved in sending the email. I was assigned to look at the problem by my immediate manager. I had not really seen this part of the application before, so its process was completely new to me; I had to learn as I went.

After some investigation I found the issue and reported back. My manager then asked me to make a change in order to fix the bug. However, I was asked to do make this change in the live version of the application, not the development version. For the non-software developers out there, this is not best practice and is potentially dangerous.

I didn't question my manager as I assumed (wrongly) that if he's willing to allow me to make a change like this to the live application, it couldn't really be that catastrophic if something went wrong, so I made the change. I asked about how we could test this and he replied:

"Client X have scheduled an email to go out to 5k clients in 10 minutes; that'll be a good test".

He then left for his lunch, and so did I. Upon returning, I was informed by the support staff that the process had failed, meaning an incorrect email had been sent to their entire database rather than a subset of their clients.

The fallout

This morning, our managing director arrived at the office and was extremely cross at what happened. He hauled both my manager and myself in to a meeting immediately and asked what happened. My manager stayed mostly silent throughout the entire meeting, leaving me to do all the talking. At the time, I didn't want to throw my manager under the bus as I wanted to allow him to admit his mistake in instructing me to do the job on a live system. However, this time did not come. The managing director then stopped me talking and issued us both with a written disciplinary notice as there is a potential to lose this client who is worth about £200k per year.

My question

I've been given the right to appeal the written notice, however I must do so within 7 days. I feel as if this written notice I have been issued with is unfair as I was only acting under direct instruction from my manager. I spoke to my colleague about it today, and she told me that she heard the entire conversation between my manager and I about testing on live data and thinks that my notice is unfair.

Do you think I have a point for appealing this decision? and if so, how would I approach it? We're a small company (10 employees), and I don't really want to make things awkward between me and my manager in the office.

Answer 1

There are numerous comments and answers criticizing what the standard practices not being followed etc but I know from personal experience that such incidents are not too uncommon in small mom-n-pop type of non-tech companies. There is usually no change management, no standardization, inadequate test/staging environments, no version control...and everyone (sometimes even the non-IT guys) have administrative access to production databases/servers. In such companies, most small enough changes go directly to production/live systems because the existing developers know the systems inside out (such systems are not too complex either). But when someone new joins, he/she is very likely to make such mistake initially when not properly supervised.

What happened is clearly that your manager took your word for it when you said you have fixed it but as it turned out, you had not fixed it. It is not your fault either because you were new to the system and your manager should have known better.

That being said, I think this could actually be a great opportunity for you to either get promoted or gain sufficient respect in the eyes of your managing director for the next appraisal. Below is what I would recommend you can do -

1. Setup a meeting with your MD (and anyone else higher up except your immediate manager) and explain to them what transpired. Don't blame your manager and don't take the blame all on yourself but do explain the facts of the case. Don't forget to make it clear that it was not your own idea to mess with the production system and that were following your manager's instructions throughout (but again without directly blaming him, a tricky conversation). If you can't do this conversation without making it look like you're backstabbing your manager then don't do it.

2. Then emphasize on the fact that the real problem here is the lack of industry best practices like change management, release management, TDD, continuous integration etc in the company. Explain how they were followed in your previous organizations and that such issues were nonexistent there. Tell him confidently that you are well versed in those practices and if he likes, you can implement them here as well. This step is is going to launch your career in this company because this will change you from the-new-guy-who-messed-up to the-new-guy-who-will-make-our-problems-go-away. Do make sure to thoroughly understand the best practices before this meeting so that you know what you're talking about, people in senior positions are usually smart enough to see through lies.

Remember that the key is to use all of the tact at your disposal to shift the focus from who did it to what the real problem is and how you're going to implement the industry best practices to make such issues go away forever (and doing it while giving the impression of being completely honest and sincere, believe in yourself and your ability to do it).

The above steps will most likely eliminate the need to worry about the written warning and stuff and you'd have placed yourself in a very favorable position with your MD. In addition, you'll have the opportunity to implement the best practices that will not only help strengthen your position but also improve things in your company across the board. It's a small 10 people shop, if you're good with your MD, you can stay there forever.

Good luck.

Answer 2

As I don't know your jurisdiction, this is my personal opinion heavily based on European influences:

• You did what you were ordered to do.
• You knew it was potentially dangerous.
• Both you and your boss were warned for this.

So yes, you acted on a direct order. However, your duty as a good employee is to warn your supervisor if he does something wrong. So in my eyes, what you need to prove is that you did. That you not only relied on your supervisor to be right, but you presented your own knowledge and your supervisor overruled you.

So if you have a mail where you tell your supervisor that you don't want to do this because it's dangerous and a reply from your supervisor where he tells you to continue despite your objections, then you definitely have a case where it's his fault alone.

If you don't have any of this, maybe because you assumed your boss would know what you know, then I guess you both deserve the warning. Your supervisor for making a stupid decision and you for not calling him out on it.

Answer 3

If you were following explicit instructions from a superior, then you have a point in appealing. However, depending on your status and experience there would have been some expectation on you to anticipate and alert your boss to his potentially risky instruction. Was it reasonable to expect you to know the likely consequences of a mistake? Had you warned him and he had insisted on proceeding, you could reasonably be considered blameless; that's not the case here and you should be prepared for some of the blame. If there's one lesson to come away with, it is that you should always follow your instincts in checking what you're asked to do if you have genuine doubts.

At the time, I didn't want to throw my manager under the bus as I wanted to allow him to admit his mistake in instructing me to do the job on a live system. However, this time did not come.

Well done on taking the moral high ground. However, it's clear your boss is not above throwing you under the bus. And here lies the greater value of an appeal - if senior management have the impression that you had the leading role in this, and you simply accept the warning while he appeals forcefully, you're likely to take the full weight of the consequences.

Answer 4

This is what's often referred to as a "career limiting move" (or CLM). Regardless of whatever else happens, you're the guy who almost cost the company a £200,000 a year client. (Or the guy who actually did, should the client take its business elsewhere).

You made two mistakes that I can see.

The first was deploying to production without a written directive from your boss - you should have sent an email outlining the risks, advising against it and requesting written confirmation of his decision to do it anyway. This is how you protect yourself when a decision like this backfires.

The other mistake was expecting your boss to simply volunteer that it was his decision when you met with upper management. Hopefully little else needs said on this note. Expecting people to voluntarily take the fall for you is ... unwise, to say the least.

You can appeal the decision, but both these mistakes will make a favorable decision that much harder now. And even if you do get one, so what? Upper management knows/believes that you are one of the two people responsible for this disaster, whether you have an official reprimand or not.

Both these mistakes are learning experiences for your future, but I would advise that regardless, you have limited your future at this company. They may or may not decide to get rid of you (and/or your boss), but even if they don't, your opportunities for advancement will be very limited. Therefore, in your position, I'd recommend trying to find another job at a different company, where you don't have a black mark against you.

Answer 5

I'd appeal and quietly start looking for a new job. No way I would want an accepted written warning on my record, never mind anything else. And it may be a prelude to termination in any case as the self-admitted scapegoat.

General rule is not to admit to anything formally without weighing up very carefully the possible repercussions. You have no 'real' idea of their agenda, but you can pretty much bank that it's not favourable to you.

Answer 6

As a developer, I believe that you handled the situation really badly and now trying to find some support that you are not to be blamed. In my opinion you should admit your mistake and either look how to improve your reputation or look for another job.

Here is my justifications for this belief:

---

I was asked to do make this change in the live version of the application, not the development version. I didn't question my manager as I assumed that if he's willing to allow me to make a change like this to the live application, it couldn't really be that catastrophic if something went wrong, so I made the change

You are a developer with 8 years of experience and you just thought that it is ok to just do stuff on prod-service which can potentially affect huge amount of users. If you thought that this is wrong, it is your responsibility as a professional to warn a person who gave you a task and explain all the consequences. It does not look that you have done so, but instead you just assumed that everyone else except of you will be responsible and everyone else agrees with this.

I would add that this

I had not really seen this part of the application before, so its process was completely new to me, I had to learn as I went.

is not a valid excuse, because in any reasonably big system there are many parts of the application that one has never seen before and even there are some parts that no one in the organization has never seen before.

---

Another part:

I asked about how we could test this and he replied: "Client X have scheduled an email to go out to 5k clients in 10 minutes, that'll be a good test"

and still after 8 years of experience you agree that testing potentially dangerous application which you admitted you had never seen before should be done on live real people. Really? Where is your hey manager, what the hell is this? We should never do this and this is one of the worst things that you can suggest me. Here is what I think we should do: xxx. And then if he really think that this is what you should do - let you give this in written form.

---

ok, and finally

He then left for his lunch, and so did I. Upon returning, I was informed by the support staff that the process had failed

so you have done two very dangerous mistakes and instead of sitting and being ready to fix everything the second after the collapse, you decided to leave (or you were expecting that everything will go smoothly?)

---

So based on this I believe that you are even more to blame here than your manager.

In the end imagine this situation: you come with one of your relatives (who is very ill) to a hospital. You are paying the doctor, so you kind of the boss. The doctor start treating your relative and you start to give him suggestions what to do. The doctor blindly follows whatever you told him to do ignoring the fact that he has 8 years of experience and what you suggest does not make sense and then you both happily go to eat.

Something terribly bad happened to a patient and the doctor says - 'you know what - I just followed what my boss said to me'.

This is very exaggerated situation but the main point is: you are the professional, you should know potential problems with dangerous stuff you do. It is your responsibility to explain them to the management. And do them only if they heard your complains and instructed you to do otherwise in written form.

Answer 7

Honestly I'm surprised you were both not fired for this. This is a rather serious breach of customer confidence in the company. In the software development industry, as you know, playing with live data is never a good idea especially if such manipulation is going to affect a wide range of customers. Affected customers may get angered and voice their anger towards the company. If higher ups get word of this, they'll go after everyone responsible, regardless of who said what. Think about the Wells Fargo scandal going on right now. 5800 people were fired. How many you think were simply following what their superiors told them?

However, I do agree that it is very hard to report such a breach in process when there are no channel to do so. Even if you do report it, it's very easy for your manager to flip it back on you.

So I sympathize with you that this is a rather difficult thing to block. For future cases I believe you can deflect this by first saying you think you identified the problem but you need to test it after you set up a development environment. Make sure you get each and every email clearly and make sure if you are asked to manipulate live data that you repeat it back with a question, "I want a clarification that you want me to change live data with something I didn't test?"

It might not be enough to protect your job but it will be enough to where if you were wrongly fired that you could have evidence to the contrary.

Answer 8

Do you think I have a point for appealing this decision? and if so, how would I approach it? We're a small company (10 employees), and I don't really want to make things awkward between me and my manager in the office.

Your behavior of trying to avoid uncomfortable feelings is coming at a high price. You got called to the carpet, and thought your manager was gonna be the virtuous knight in shining armor and rescue you, but that didn't work. Your manager probably knew better than to be adjusting production code, but took a shortcut. By that, there was already a red flag as far as integrity was concerned. So maybe you were expecting too much in the meeting.

Now you're still concerned that there's going to be some discomfort, even though you actually did nothing wrong, if you tell the truth. Your manager apparently has a better stance at dealing with negative emotions than do you. You're putting his feelings ahead of your own ability to bring home a paycheck. To save your own tail, you're going to have to get wise, emotionally.

Here's where this can go, if things sour: no job, late notices in your mailbox, and an empty refrigerator. Speak up for yourself.

Answer 9

Appeal it. You may not win, but you can at least get your story out.

Do so humbly, explain that you were told to do what you did, and ask what you should have done differently. Tell them that you take this very seriously and want to ensure this not happen again, regardless if there be a warning on your record or not.

In the mean time, and forever after this point, document everything. If a manager pulls you aside and says "do this". The first thing you should do when you get back to your desk is to write an email that says:

"As per our conversation earlier today, you instructed me to do ABC. Should anyone else be informed before I start work on this, and are there any concerns/dependencies/required permission I should obtain before beginning work?

I just want to make sure everyone is on the same page. Thanks in advance for your clarification"

This is how you can document even side conversations and make sure that is clear in advance that you are not a loose cannon.

As I used to tell my reports "Anything before something happens (deadline, mistake, et cetera) is expressing a concern, anything after is an excuse which nobody will listen to. Be full of concerns, and document them.

Answer 10

You don't want take the main blame retroactively and others discussed that. However, you have to be aware that there most certainly is one thing that you and nobody else are to blame for: you installed an important change on a live system about to make a large transaction and then, as the person responsible for the technical execution, went to lunch when your manager did and let the shit hit the fan in your absence.

And that after this was called a "test case" and thus there was no full expectation of success.

That's like a demolition master setting the timer on the explosives for a skyscraper and then leaving for lunch, in the expectation that he'll return to a razed building.

That part of the incident is really topping off the recklessness of the whole event and rightfully belongs on your record and is not something you can reasonably expect to be cleared of. So the question is how much you hope to improve your situation by getting other stuff corrected.

Answer 11

I feel strongly that you and your boss both deserve the warnings you each got for deploying a hotfix live into production and then going to lunch.

The fact that you were ordered to deploy the hotfix live is your bosses mistake (and possibly yours for not pushing back strongly - there are times where my bosses might ask me to do something that I know is dangerous and I'd be expected to refuse and point out why as the subject matter expert for that area). But fine, we'll give this one to your boss, this is where he earned his warning.

The fact that you then went for lunch when you knew there was a live run about to take place with client data is absolutely mind-boggling. I'm sorry but this is the time at which your written warning was truly earned, and I don't see how appealing will help. You needed to be around to take responsibility and ownership of any issues that occurred with this fix and you fell down on this by going to lunch at the exact wrong time.

Answer 12

Your manager's duty is to determine what needs to be done; set priorities, make tough decisions and the like.

Your duty is to know how to do what needs to be done, to keep in mind best practices, to calculate the trade-offs... and communicate this knowledge to your management and help them take the right decisions.

You failed to acknowledge a dangerous situation.

You failed to communicate this to your manager.

You mention that you didn't know much about this part of the system.

However you have been working for 8 years in a 10 people company.

If I were your manager I wouldn't expect you NOT to know what you are doing.

• If you didn't mention your manager your lack of knowledge in this part of your system,
• and if you didn't mention the potential risks on making changes on live and not in development to test them

Then, I am sorry - it is your fault.