Recently joined a team as a junior dev, and learned they plan to have a certain amount of failure. Is this normal?

Question

A couple months ago I joined a team as a junior dev and I recently got added to the support rotation. This week is my week and my senior told me that I should go home early Wednesday as there would be a failure around 10PM. I asked him how he know and why we did not just fix it now and he told me that if things don't fail every 2-3 months in a way that clients see and complain about, management will wonder why they pay for an extra developer to do support work.

I really do get why they do this. They wouldn't buy us a new hard drive for the server until it crashed and was costing 5K an hour in penalties and clients were screaming at our VP a month ago. They tell us to pirate our IDEs or just use Notepad++. They won't pay for a bug tracker, so we just let bugs flood production to be the little explosions which get us attention.

I have other recent grad friends who are doing similar things, whether at Microsoft, Amazon, or small startups without any technical leadership. Is this normally how luddites/business suits are dealt with? Whether it be Boeing or NASA or just the anecdotes from friends, it seems very common. I got into engineering to avoid conflict. Am I going to spend a ton of time fighting with biz school overlords?

Answer 1

You cite the Challenger disaster and the more recent Boeing debacle - but you should also read up on where NASA, Boeing, and many other organisations' management got things right over the years (which happens a lot more). As for the Challenger disaster, the investigation and inquiry led to significant changes in management process.

As for anecdotes, every developer has a horror story to tell about a nightmare employer or client, we like to talk them up because the other stories, where management listened to advice, are just plain boring.

As for where you're at - it sounds like a very bad organisation - especially with regards to suggesting you torrent licenced software and not undertaking preemptive maintenance. Having devs on support is also not optimal (although you don't specify what kind of support), and scheduling "failures" is likely to end up with your senior getting fired if he's found out. There are companies out there that are like this - my friend used to work for one. But they are not the norm.

That said, there will inevitably be a time where you have to justify expenditure to the business people - and it helps to learn their language. Money! If you can show that it's cheaper to preemptively replace hardware on a regular schedule instead of acting service penalties at the inevitable failure, they just might listen. On the other hand, they might not, since people are people and not always smart.

Answer 2

Keep in mind: Failure usually/eventually destroys the suboptimal.

Do companies operate like you're describing? Yeah, some do. But a company with such a terrible relationship with IT generally dies. Which makes sense - if you have one company that works as yours does, where IT sabotages the experience of the clients, and a second company that has a functional IT relationship... which do you think will have a better chance of surviving? The one where IT doesn't spend effort trying to make the customers miserable.

Do managers/leads operate like you're describing? Yeah, some do. But a manager who arranges failures as a way of gaining visibility/money will start to look awfully terrible compared to one that works towards positive outcomes.

To be honest? I almost wonder if your current IT department understands about delivering value to the non-IT sections of the company. A question of whether you need an extra developer shouldn't be "Hey, we had 20 outages - if we lose a person, we'll have even more!" It should be "Look at what all we managed to build for you last year - the new Sales portal, the customer retention tracker, and the proof-of-concept for machine learning on customer habits. If we lost a person, we wouldn't be able to deliver as much." Part of me wonders if your company should just fire the IT Director.

Answer 3

Engineers are not really respected by businesspeople.

This Big Bank Theory skit is a pretty good representation of how many business people think of engineers except they drop the "noble" and prefer "expensive".

I could see this as normal wherever technical people are not represented in management in any meaningful way and need to get anything done. I work for the government so the lack of pressure prevents it from manifesting here, but I have friends in the corporate enterprise dev groups where their team leads do this. They can barely get meetings otherwise.

Probably the most common version of this is inventing a reason to develop the next tech thing in a new tech stack. https://completedeveloperpodcast.com/episode-172/.

The engineers allowing errors are just saving themselves either a futile conversation with a non-technical beforehand or greater difficulty meeting expectations later as "they don't need the resources." Is it technically right? No. Do I blame them? Not at all.

The further away engineering is from the core offering or the decision-making class of the company, the more of this you will see. I would bet that you are in a corporate enterprise environment.

A good article on Boeing, where management literally ran away from the engineers and everyone else making the planes: https://www.theatlantic.com/ideas/archive/2019/11/how-boeing-lost-its-bearings/602188/

P.S. I feel ya on the Agile.

Answer 4

As a developer, I can identify with Matthew Gaiser's answer and understand the frustration that your question clearly shows. But with over a decade as a Director of service delivery for global organizations, I believe that your question highlights much bigger problems within the organization.

TL;DR

If you can't help solve the problem you are part of it.

Planned Failures

Most service providers or SaaS providers within the B2B market are contractually obligated to customers with Service Level Agreements (SLAs) or Operational Level Agreements (OLAs) that often dictate penalties for failing to maintain the guaranteed uptime/availability of the service or application. Willingly and purposefully causing failures and downtime, despite your frustrations, is in essence willful negligence on your part and depending upon the jurisdiction could result in criminal and/or civil penalties.

Failed Hard Drive

Failure of management to approve the cost of replacing a hard drive that surely costs less than the penalties is frustrating at best. It could be the result of not properly communicating the risk or an indication of severe financial difficulties; without additional information we have no way of knowing.

But ultimately you have a more serious issue that needs to be addressed. Why is your hardware setup and configured in such a way that a single hard drive failure results in such a catastrophic outage?

Torrent's and Pirated Software

The use of pirated software is illegal, even if you were directed to do so by your employer. It also presents significant security concerns and greatly increased the risk to your systems, information, and intellectual property. Don't use pirated software.

Depending upon the language and your platform there are any number of free and/or opensource IDE tools that are very high quality and can get the job done without additional cost.

Bug Tracker

They won't pay for a bug tracker, so we just let bugs flood production

I'm sorry but I have to blunt here. This is a piss poor excuse to justify your own lack of professionalism. Nothing more.

There are a number of opensource/free bug trackers readily available that can be easily implemented at little to no cost. Many of them will integrate with the free/opensource IDE's that are available.

Answer 5

Run far far away and do it fast!

To be blunt and honest, this is a company that you should get away from as soon as you can. You are learning all sorts of bad business practices that will not set you up for success in the rest of your career.

• They are asking you to do illegal things like pirating software, which could get you in legal trouble as well as stain your professional reputation if future companies find out about it
• Your IT manager is asking you to do immoral things like creating fake problems to get attention and money from the rest of the business rather than justifying expenses through legitimate business needs and by delivering value to the company
• Your department is behaving unprofessionally by intentionally allowing bugs to go out to production because they are upset that they don't have proper bug tracking. This is like a kid taking his ball and going home because the other kids won't capitulate to his demands. It is unprofessional and a horrible lesson to learn from a job

While there are companies like this, it is not normal or acceptable.

You will certainly have to spend a portion of your time and career dealing with non-technical people both within your company and within your client/user base. Those are skills that need honing just as much as the technical parts of your job. Don't let this job teach you that these practices are the right way to do that, they absolutely aren't.

Ideally find a job where the management has at least some technical knowledge so that you have fewer of these problems, but if not then find a job that teaches you the right way to handle these situations professionally.