I submitted an unapproved feature, got in trouble, now what?

Question

TLDR: I pushed (broken) functionality without approval that ended up being caught by QA, and my boss is (rightfully) upset. What should I do?

I work as a developer on a digital product.

Recently, under tight deadline, we submitted a build to be reviewed by our QA team, and a report came back last night that they found a bug in the product. This bug was functionality I added, without approval, as a test for a future version of the product.

I had put this functionality behind a sort of passcode that had to be entered by interacting with an in-product element in a certain sequence, with the assumption on my part being that only I would be able to trigger it, and no one else would see it. I later realized that the code I initially put in would actually trigger the unapproved functionality even if the sequence was included in a larger sequence, meaning a series of random interactions would have a possibility of triggering the funcionality (though I thought it would still be rare). I made a fix to only allow the precise sequence, but that code did not make it into the build sent out.

A few days after that (yesterday), after QA tested the product, they accidentally triggered the sequence and reported the resulting functionality as a bug. My boss showed it to our creative director, who was also able to trigger the functionality. He then spoke with me about the matter this morning.

When my boss spoke with me about this, he expressed several concerns:

1. I added this functionality without seeking any approval
2. I created a "fix" that did not remove the functionality
3. In code commits I referenced it as being an "easter egg", implying the intent to push this live
4. My functionality is both broken and off-brand
5. This would have gone live if QA didn't catch it
6. Our timeline (and related marketing/featuring opportunities - $$$) are jeopardized as a result of having to make another build now
7. Extra QA costs are now necessary
8. I wasted time that should have been spent on higher priority bugs and features

In addition to the above, he said he would now have to speak with his own boss to decide what they would do about the matter. As it turns out, he approached me shortly after, asking me if I would be able to make the smallest possible code change to remove the functionality, possibly being able to bypass a full QA sweep, and salvage our release window. I immediately pushed a one-line change that removed the functionality, showing him the code that I changed to do so. He thanked me, but I have not spoken to him since (earlier today).

Though I realize the error in my judgement now, at the time my actions were informed by this thinking: Unfortunately, many times my boss and director will see a partially-completed feature (by myself or another dev) or hear an idea described verbally, and reject the feature or functionality, due to what I feel is a lack of vision for what a feature could be in its final form. When I have instead taken the approach of presenting something more polished to them, it is nearly always approved. We recently discussed a feature like the one under question, and so I put in an hour of work to just get in the rough functionality, to use as a demonstration. I thought with my "passcode" wall, no one would accidentally see it.

I now realize how serious this could be. In the past, even if I worked on things so that I thought I could prove their worth, I always showed things I worked on and presented them for final approval, far before a final build. I realize I stepped over the line by allowing something to go live, even if I thought I was keeping anyone from seeing it accidentally.

I think this could be something my boss might fire me over. Though I am one of the stronger performers on the team, it might be that this breaking of trust is enough to warrant my termination.

I do not want to lose this job. I enjoy my work, it is challenging and creative, and I am grateful for the pay and flexibility. I sent a message to my boss apologizing, saying I realize the severity of my actions and that I broke trust and will never do something similar again, going off-spec without approval, etc. He has not responded yet.

Any advice as to what to do at this point would be greatly appreciated! Thanks in advance.

Answer 1

In the future, when working on stuff like this, check it into a branch rather than the master. Or go low tech and save it elsewhere on your personal hard drive. Unless you're in a crisis the problem wasn't the time spent- it was the risk of something unknown, untested, and potentially breaking going out.

What to do- talk to your boss. Explain to him that you understand what you did was wrong. Tell him why, so he knows you understand. Propose a process to make sure it never happens again (the process is code reviews. With code reviews this would never get through. Unless you have code reviews and circumvented them, in which case the answer is a technical lock preventing merges to master unless its been reviewed- and in that case you're in much deeper shit, because circumventing code review looks shady). And accept you're going to be in the doghouse for a while.

How much trouble you're in depends on what your job is and what the functionality did. If you work in something like defense, banking, etc- you're done. If you make a website and your code has nothing to do with accessing other people's accounts or payment info, you're likely ok. Not enough info to tell here.

Answer 2

What you did was bad, in fact I'd go so far as to say appalling, but then I'm sure you don't need me to tell you that. And you certainly don't need an internet randomer having a go at you.

What to do now?

Pretty much stay along the lines you have already been following:

1. Do apologize unreservedly, explain that you understand that what you did was wrong and that you know why. And, perphaps most importantly emphasize that this will never happen again.

2. Do not attempt to justify what you did. Using initiative and roughing out PoC's to demo ideas is not in of itself a bad thing, and I've always encouraged developers working for me to do just that. But when you are working to a tight release deadline is not the time and submitting it into the build for live is most definitely not the place and attempting to explain your thinking along those lines to me (were I your manager) in this case would just irritate me further.

3. Do not attempt to spread any blame. As other answers have pointed out this highlights a potential weakness in your organisation's code review process and I'm not saying they are wrong but you are not the person to be saying that to them right now. Doing so would only come across as you saying "I shouldn't have been able to do this 'crime', you should have caught me", which is immature at best and victim-blaming at worst.

4. Do apologize to the QA team as well, you not only wasted their time you were attempting to slip something past them which is fundamentally dishonest. You're supposed to be on the same "side"!

Assuming the release schedule can be salvaged without too significant a cost to the business and your track record is otherwise very good with this incident being out of character then you might survive it but I expect it would take quite some time of you being whiter than white to rebuild the organisation's trust in you. Good luck!

Answer 3

Well, you've learned the hard way that you always need to seek advice before including a functionality.

The first thing will be to tell your boss about this, to show you fully understand how serious was that matter and won't do it again. Openly admit a mistake is a really professional thing.

Next step, win back the trust of your boss. You might have to lay low for a while. But then you can show him that his opinion matters, ask him for advice, even for decisions that are not as big as putting a new functionality into production.

You messed up, you know why, be patient and good at your job and you should be good.

Answer 4

Message to your boss: Boss, your code review process is totally and utterly broken. Your processes should be set up so that a developer cannot add code that will be added to production without the code itself being reviewed and accepted by another developer. So it's your fault as much as the developer's fault.

Message to you: That's not how it works. You can't just go ahead and do things on your own. If you have an idea then go to your boss and tell him about it. And if it's a good idea, he or she will order you to implement it, and everything is fine. But you need to realise that what you think is a good idea and what the company thinks is a good idea can be very different. The first question to ask for every idea is: Will we make more money? Will we sell more products? Will customers be happier and give us better reviews? On the other hand: Will we need to have documentation? Will it create problems for support? Will it make future development harder? That's all things that your boss would consider. It's not your job to consider these things, but you have to go through your boss.

The good news: This isn't something you usually would be fired for. The problem was caught, no damage was done (you were lucky there, buy the tester a beer when you have a chance). This was an educational moment for you. Your boss my very well come down harder on you than necessary to make the educational part of this stick better, without any actual consequences for your career. If this had gone out to customers, that would be different.

@Lilienthal: If the boss is told that his development process, for which he is responsible, is broken, that is never not timely advice.

Answer 5

First, the bad news: You acted very foolishly and impulsively.

You know this, no need to say any more.

Now, the good news: You probably won't be fired.

Give it a few days, apologize and detail to your boss the following

1. WHAT you did
2. What was wrong with it
3. What damage it could have caused
4. Admit to not seeing the repercussions at the time.
5. Ensure your boss that it will not happen again
6. Express your regret
7. Move on

If I were your boss, I would likely not fire you, but I would watch you closely for the foreseeable future. I'd probably also have a bit of a private chuckle to myself when the dust settled, then I'd think about how to channel your creativity.

Programmers are a mischievous lot, and that's fairly common knowledge so what you did wasn't completely out of the realm of expectations. Unfortunately, times have changed and this mischief is frowned upon these days.

Rebuild your manager's trust, and in the future, learn to advocate for your changes/features. Be able to quantify what you're pushing and go by the outlined procedures. Demonstrate in a demo, but never try to sneak anything into production again.

Answer 6

First of all: breathe

Let's unwind the chain of events:

I think this could be something my boss might fire me over. Though I am one of the stronger performers on the team, it might be that this breaking of trust is enough to warrant my termination.

It could lead to your firing but it hasn't. In fact, he gave you a chance to rectify your mistake in the short-term. He even gave you positive feedback.

In addition to the above, he said he would now have to speak with his own boss to decide what they would do about the matter. As it turns out, he approached me shortly after, asking me if I would be able to make the smallest possible code change to remove the functionality, possibly being able to bypass a full QA sweep, and salvage our release window. I immediately pushed a one-line change that removed the functionality, showing him the code that I changed to do so. He thanked me, but I have not spoken to him since (earlier today).

A little technical side-note:

I thought with my "passcode" wall, no one would accidentally see it.

Look into proper feature toggling. A lot of frameworks support it. Executed properly, this will help you quickly disable buggy features live while you fix the underlying issue and this mistake might very well be a turning point for your software team.

Unfortunately, many times my boss and director will see a partially-completed feature (by myself or another dev) or hear an idea described verbally, and reject the feature or functionality, due to what I feel is a lack of vision for what a feature could be in its final form. When I have instead taken the approach of presenting something more polished to them, it is nearly always approved. We recently discussed a feature like the one under question, and so I put in an hour of work to just get in the rough functionality, to use as a demonstration.

This right here is one of my biggest weaknesses (not completing ideas often enough). Recognise this weakness and figure out how to combat this. Force yourself to not work on new exciting stuff until you've worked through a certain amount of unfinished issues.

Lastly, ask for a sit-down with your boss and say you'd like to talk. Then lay out how you are gonna work on not repeating this kind of mistake. Make a list of actionable points (for instance): - look into feature toggling - look into completing work on issues before submitting - Remember to also talk about that your feelings (insecurity about your job). Any good, sympathetic person (I'm assuming your boss is one) will surely not hold you accountable for being insecure after making a mistake.

Good luck!