7

A couple months ago I joined a team as a junior dev and I recently got added to the support rotation. This week is my week and my senior told me that I should go home early Wednesday as there would be a failure around 10PM. I asked him how he know and why we did not just fix it now and he told me that if things don't fail every 2-3 months in a way that clients see and complain about, management will wonder why they pay for an extra developer to do support work.

I really do get why they do this. They wouldn't buy us a new hard drive for the server until it crashed and was costing 5K an hour in penalties and clients were screaming at our VP a month ago. They tell us to pirate our IDEs or just use Notepad++. They won't pay for a bug tracker, so we just let bugs flood production to be the little explosions which get us attention.

I have other recent grad friends who are doing similar things, whether at Microsoft, Amazon, or small startups without any technical leadership. Is this normally how luddites/business suits are dealt with? Whether it be Boeing or NASA or just the anecdotes from friends, it seems very common. I got into engineering to avoid conflict. Am I going to spend a ton of time fighting with biz school overlords?

  • @SolarMike we at least have that for our networks... – AgileSucks Apr 14 at 7:18
  • And: Am I going to spend a ton of time fighting with biz school overlords? Yes, you will. Does not even matter which job you are in! – Daniel Apr 14 at 15:28
  • 4
    ...he told me that if things don't fail every 2-3 months in a way that clients see and complain about, management will wonder why they pay for an extra developer to do support work... Hopefully your description is not accurate or lacks details that would clarify that nobody is actually causing failures (which necessarily amount to damages to customers) on purpose, because the way it is worded now sounds like that to me, it's describing a fraud, something illegal, not to mention evil to the clients. It's also describing a hellish workplace though, which you should try your best to leave. – SantiBailors Apr 14 at 16:06
  • 1
    @SantiBailors Agreed. We have irregular "failures" at my office due to infrastructure limitations (which will vanish when an upgrade project is completed) The cost of applying a workaround each failure for the remainder of the lifespan versus the cost to develop and test a 'permanent fix' is such a vast gap that, even if the upgrade project takes 5 time longer than expected, the workaround will still have been the cheaper option. That it reminds management to keep funding the upgrade instead of trying to sacrifice it for short-sighted budget savings is genuinely and entirely a side-effect – Chronocidal Apr 14 at 16:21
  • As an engineering leader, if I found an individual engaging in this kind of unethical behavior, they'd be terminated immediately. – Joel Etherton Apr 22 at 22:39
9

You cite the Challenger disaster and the more recent Boeing debacle - but you should also read up on where NASA, Boeing, and many other organisations' management got things right over the years (which happens a lot more). As for the Challenger disaster, the investigation and inquiry led to significant changes in management process.

As for anecdotes, every developer has a horror story to tell about a nightmare employer or client, we like to talk them up because the other stories, where management listened to advice, are just plain boring.

As for where you're at - it sounds like a very bad organisation - especially with regards to suggesting you torrent licenced software and not undertaking preemptive maintenance. Having devs on support is also not optimal (although you don't specify what kind of support), and scheduling "failures" is likely to end up with your senior getting fired if he's found out. There are companies out there that are like this - my friend used to work for one. But they are not the norm.

That said, there will inevitably be a time where you have to justify expenditure to the business people - and it helps to learn their language. Money! If you can show that it's cheaper to preemptively replace hardware on a regular schedule instead of acting service penalties at the inevitable failure, they just might listen. On the other hand, they might not, since people are people and not always smart.

| improve this answer | |
  • Here's a positive story: We had a major FUBAR on an invoice processing system, resulting in 30,000 electronic invoices to be processed by hand (10,000 a day, failure over the weekend). I spoke to the CEO and recommended. No more deployments on Friday. Latest Wednesday noon. No compromise. Became a company rule for the last 10 years. Good stories are boring, but learn from the bad ones. – Nelson May 25 at 8:42
6

Keep in mind: Failure usually/eventually destroys the suboptimal.

Do companies operate like you're describing? Yeah, some do. But a company with such a terrible relationship with IT generally dies. Which makes sense - if you have one company that works as yours does, where IT sabotages the experience of the clients, and a second company that has a functional IT relationship... which do you think will have a better chance of surviving? The one where IT doesn't spend effort trying to make the customers miserable.

Do managers/leads operate like you're describing? Yeah, some do. But a manager who arranges failures as a way of gaining visibility/money will start to look awfully terrible compared to one that works towards positive outcomes.

To be honest? I almost wonder if your current IT department understands about delivering value to the non-IT sections of the company. A question of whether you need an extra developer shouldn't be "Hey, we had 20 outages - if we lose a person, we'll have even more!" It should be "Look at what all we managed to build for you last year - the new Sales portal, the customer retention tracker, and the proof-of-concept for machine learning on customer habits. If we lost a person, we wouldn't be able to deliver as much." Part of me wonders if your company should just fire the IT Director.

| improve this answer | |
  • 1
    given that the op complains that "They tell us to torrent our IDEs" i'll wonder if it's a wise move to stay at this company. being told to become a criminal is a huge red flag, as well as a company not wiling to buy the basic tools for his workers. sounds more like the rest of the company has no clue about the value IT brings to the rest of the company ... or, more precisely, the rest of the companies management has no clue. – d_hippo Apr 14 at 8:32
  • The better product scenario that you're describing here seldom applies, companies will not have competing managers for the same roles neither will customers for the same product. This might apply if the deciding factor between manager or product is the number of outages, or if there is no tolerance for more outages. Yes it might be detrimental but not as a general rule for all cases, but specific to some. The OP is in a better position to know for his case. – DrDread Apr 14 at 14:41
3

Engineers are not really respected by businesspeople.

This Big Bank Theory skit is a pretty good representation of how many business people think of engineers except they drop the "noble" and prefer "expensive".

I could see this as normal wherever technical people are not represented in management in any meaningful way and need to get anything done. I work for the government so the lack of pressure prevents it from manifesting here, but I have friends in the corporate enterprise dev groups where their team leads do this. They can barely get meetings otherwise.

Probably the most common version of this is inventing a reason to develop the next tech thing in a new tech stack. https://completedeveloperpodcast.com/episode-172/.

The engineers allowing errors are just saving themselves either a futile conversation with a non-technical beforehand or greater difficulty meeting expectations later as "they don't need the resources." Is it technically right? No. Do I blame them? Not at all.

The further away engineering is from the core offering or the decision-making class of the company, the more of this you will see. I would bet that you are in a corporate enterprise environment.

A good article on Boeing, where management literally ran away from the engineers and everyone else making the planes: https://www.theatlantic.com/ideas/archive/2019/11/how-boeing-lost-its-bearings/602188/

P.S. I feel ya on the Agile.

| improve this answer | |
  • 4
    Wow in 20 years of work experience I have never seen an employee actively sabotage the business. Am I just lucky or is it really that bad on the other side of the pond? – Daniel Apr 14 at 15:33
  • @Daniel it depends on how you define "actively." Most of the time the devs have requested time/permission/tools/hardware for a fix and get denied. The devs treat that as management being fine if it fails and then they basically wait for instructions to fix it when it does. These aren't bugs they are introducing. This is also specific to a certain type of environment. It happens in corporate organizations where development is a support function, its a fight to get $10 for VM cloud space, deadlines are firm, and everything is fragile. Remove one of those things and it doesn't happen. – Matthew Gaiser Apr 14 at 16:30
2

As a developer, I can identify with Matthew Gaiser's answer and understand the frustration that your question clearly shows. But with over a decade as a Director of service delivery for global organizations, I believe that your question highlights much bigger problems within the organization.

TL;DR

If you can't help solve the problem you are part of it.

Planned Failures

Most service providers or SaaS providers within the B2B market are contractually obligated to customers with Service Level Agreements (SLAs) or Operational Level Agreements (OLAs) that often dictate penalties for failing to maintain the guaranteed uptime/availability of the service or application. Willingly and purposefully causing failures and downtime, despite your frustrations, is in essence willful negligence on your part and depending upon the jurisdiction could result in criminal and/or civil penalties.

Failed Hard Drive

Failure of management to approve the cost of replacing a hard drive that surely costs less than the penalties is frustrating at best. It could be the result of not properly communicating the risk or an indication of severe financial difficulties; without additional information we have no way of knowing.

But ultimately you have a more serious issue that needs to be addressed. Why is your hardware setup and configured in such a way that a single hard drive failure results in such a catastrophic outage?

Torrent's and Pirated Software

The use of pirated software is illegal, even if you were directed to do so by your employer. It also presents significant security concerns and greatly increased the risk to your systems, information, and intellectual property. Don't use pirated software.

Depending upon the language and your platform there are any number of free and/or opensource IDE tools that are very high quality and can get the job done without additional cost.

Bug Tracker

They won't pay for a bug tracker, so we just let bugs flood production

I'm sorry but I have to blunt here. This is a piss poor excuse to justify your own lack of professionalism. Nothing more.

There are a number of opensource/free bug trackers readily available that can be easily implemented at little to no cost. Many of them will integrate with the free/opensource IDE's that are available.

| improve this answer | |
2

Run far far away and do it fast!

To be blunt and honest, this is a company that you should get away from as soon as you can. You are learning all sorts of bad business practices that will not set you up for success in the rest of your career.

  • They are asking you to do illegal things like pirating software, which could get you in legal trouble as well as stain your professional reputation if future companies find out about it
  • Your IT manager is asking you to do immoral things like creating fake problems to get attention and money from the rest of the business rather than justifying expenses through legitimate business needs and by delivering value to the company
  • Your department is behaving unprofessionally by intentionally allowing bugs to go out to production because they are upset that they don't have proper bug tracking. This is like a kid taking his ball and going home because the other kids won't capitulate to his demands. It is unprofessional and a horrible lesson to learn from a job

While there are companies like this, it is not normal or acceptable.

You will certainly have to spend a portion of your time and career dealing with non-technical people both within your company and within your client/user base. Those are skills that need honing just as much as the technical parts of your job. Don't let this job teach you that these practices are the right way to do that, they absolutely aren't.

Ideally find a job where the management has at least some technical knowledge so that you have fewer of these problems, but if not then find a job that teaches you the right way to handle these situations professionally.

| improve this answer | |

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .