How to deal with half my colleagues overriding development processes under the slightest pressure?

Question

I'm a software developer. My team has a variety of development processes that code is technically supposed to go through to get to the master branch. Things like unit testing and code review.

Under the slightest amount of pressure from any authority figure (product owner, intermediate developer, scrum master, a desire to finish something before standup/sprint planning, even a random salesperson who claims something is "urgent") they will skip that and force push their fix to master to get it out into production. Our boss agrees that we shouldn't be doing this but he doesn't want to have to fight with people constantly so he just lets it slide and tells me to tell the other developers to push back. 80% of code is now going out without following the process.

The other developers view of the situation is that they will probably be here for another year at most so letting the code rot is cheaper than daily arguments about process with various people who don't value careful engineering.

What can I do about this?

Answer 1

You basically need the organization to value it as a whole.

I was with you a few months ago. I am now one of those developers you are frustrated with.

The reality is that people have certain timelines in mind and those never change. You demo something to them and then they are "where is it? where is it?" And they will do that every single time. That is on top of people who are concerned about keeping things moving along. Organizations also tend to value certain things and those values drive how things are done.

The conversation usually goes like this:

Person: "Hey, where is that feature you showed me yesterday?"

Me: "It is awaiting code review."

Person: "Well, we need it to QA/fix production issue/have it in the sprint demo/for client meeting tomorrow"

Me: "It is behind the thing you asked me about yesterday in the queue."

Person: "Well, we need it to QA/fix production issue/have it in the sprint demo/for client meeting tomorrow"

Me: "I will see what can be done."

Person (an hour later): "Any update? We need it to QA/fix production issue/have it in the sprint demo/for client meeting tomorrow."

After months and months of that, git push is a heck of a lot easier to do. Especially since as far as they are concerned, it is urgent, so they are highly motivated to get it. In many ways they are right as the deadlines are real and not something they can control. So even from the perspective of being a business unit, it is probably the correct decision.

For processes to survive, the organization as a whole (or at least the entire business unit) has to value them. Your organization clearly doesn't. Does it result in more bugs? Probably. But people outside software have come to accept bugs as just something that happens, so preventing them is often not the leading priority.

It is a question of trade offs, both for the organization and for the individual developers.

If you want to fix this, you basically need to convince sales, the Scrum master, and the product owner that there is value in not bypassing this process. They probably view it as bureaucracy.

Answer 2

Our boss agrees that we shouldn't be doing this but he doesn't want to have to fight with people constantly so he just lets it slide and tells me to tell the other developers to push back. 80% of code is now going out without

He's asking you to do his job. Thoroughly unprofessional. This should not be a constant fight. This should be an absolute law and the fighting would stop after a written reprimand or two.

There's really not much you can do in this situation. You and the other developers that care could try peer pressure, but it doesn't seem like there's enough that care enough, or haven't (understandably) given up, to make a difference.

I'd honestly start looking for another job

EDIT:

Another option, if you feel like you've tried everything with your boss, would be to go over your bosses head to their boss and seek to address this further up the chain. This would need to be done carefully, and possibly anonymously, as going over your bosses head can have serious repercussions.

Answer 3

Right now: Do nothing. Everything is fine, nothing is broken.

Next time a production-breaking bug occurs: Scream from the top of your lungs (not literally) about how this bug could have been avoided if we had had testing to catch it. Explain how careful testing and taking time is meant specifically to avoid this type of situation. Quantify how much money the company lost and how much downtime the service had because of a bug which was not caught but could have been if the developers were allowed more time to be more careful.

Management is always more open to a change in process when they see the value first-hand and immediately. If you talk in the abstract, like "well, we really should have testing, because someday we might have a problem somewhere that might take down our servers", nobody cares, because as likely as it could happen, it could also not happen, and right now it's not happening so nobody cares. However, it will certainly happen eventually, and that's when you can point to it as a pain point and show the value immediately, not in the abstract.

Of course, management could come back and say "well, if you guys were better developers then you wouldn't make bugs and you wouldn't need testing". That's the point at which you brush up your resume and find another job. Every developer makes mistakes; there is no developer who has never shipped bugged code, and it's the company's responsibility to give the developers time to make sure their code is as bug-free as possible.

Answer 4

You've misdiagnosed your problem.

What you're seeing with bypassing of standards/reviews/etc? That's a symptom of your problem.

Your actual problem is the combination of two things:

• Your boss not wanting to be confrontational about things
• Your coworkers viewing the job as temporary and only doing the minimal

Your boss has effectively delegated confronting the business area to your coworkers... and your coworkers are just going with the flow until they find another job. I'd be very surprised if standards are the only symptom from that. Are your priorities are dictated by whatever higher up screams the loudest, not what actually helps the business the most? That's not a separate problem from your question - that's something that also stems from that combo. Etc - there are likely dozens of problems, big and small, that crop up from those two factors.

Realistically, you can't fix this problem. Your best shots would be:

• Getting your boss to start doing their job, or get them replaced by someone who will.
• Making the job atmosphere enjoyable enough that your coworkers viewed it as a career.

Answer 5

For salespeople who need a feature in their demo, you could setup a demo branch and demo server. Just push whatever they urgently need and then merge it back to the dev branch and eventually master branch once unit tests and code review are done.

Skipping process to get something in before the end of the sprint or before the standup is silly and the short-term gains will be offset by having to fix something in production. Your team needs to come to an understanding of the value of tests and code review, and you might also have to revise your story point estimates if there's a rush to commit unfinished stuff before the end of the sprint.

Answer 6

In cases where outsiders start meddling in your process, simply stop explaining your process to them. Every snippet of information you give them, gives them a new hook to argue why you should/shouldn't do something.

In a company where refactoring was constantly skipped due to "urgent" deadlines (I use quotes because everything was always top priority with no exceptions), I simply stopped mentioning refactoring as a separate (and thus individually skippable) step and started counting refactoring as development work that was part of the estimate.

Instead of "2 days development, 1 day review/refactoring", which would always garner the same reaction from management ("I need you to release after 2 days, we don't have time for refactoring"), I instead said "3 days development" and did not break it down any further. Management lost the ability to argue what parts of my job I could skip just because they don't personally care about it.

Refactoring and code review, from a short-term management perspective, is a "waste" of time that could be spent on the next billable item. But it dramatically improves developer quality of life, which reduces developer burnout and people quitting, which dramatically improves the long term output of the dev team.

In companies where code quality and developers leaving in under a year is a constant issue, it's (in my experience) almost always attributed to management that meddles in development processes they don't value or understand the value of. I've worked at several companies like these.

Some managers understand the importance of the quality of life of their employees, and some managers either don't or don't care - either way the outcome is the same. When dealing with managers who fall into the latter category, I am always frugal with specifics so they don't meddle where they shouldn't be meddling.

Answer 7

This battle only needs to be fought once if you convince your boss and enough of your colleagues to set up a system of permissions that simply does not allow for this.

We use GitHub, but other services have similar options to only allow merging to the main branch after code has been approved by a code owner. Naturally, only those who take the process seriously should be code owners.

Once established, this will become a new normal. Certain processes are best not left to chance.

The main arguments I would make to a manager as to why code reviews should be enforced:

• code review is amongst the most effective measures to prevent bugs. I personally find them more effective than tests (though one should have both!). One good developer can prevent the worst from a number of less experienced or motivated developers
• it takes only one serious bug to cause potentially severe loss of functionality and/or data. Even worse, in a way, is data corruption, which may go undetected for a while and render recovery procedures like backups practically useless. This of course depends on your product.
• bugs are likely going to incur a direct cost to the business in terms of lost revenue and/or customers (again, depends on the product, but few can "afford" to be riddled by bugs)
• as a bonus, reviews are a great training tool

Answer 8

End users (sales, customer support, customers/clients/partners, etc.) should not have direct access to the dev team in general. (If the secretary, salespeople, or customer support triage is calling/emailing the developers directly, this should be addressed and they should be contacting the business side interface for the team...aka the gatekeeper.)

The dev team should direct any inquiry as to the status of a fix/change/feature to the gatekeeper of the team (tech/team lead, BA, PM, PO, whatever).

Since it is impossible to isolate a dev team from the rest of the organization altogether, it is important that the gatekeeper isn't a "yes man", takes pride in their work, and understands the concept of "haste makes waste".

If you are doing an agile approach to development with sprints/retrospectives, as part of the dev team you can bring this up in the retrospective. "We had a lot of PRs get pushed through without sufficient testing and verification, we need to work on this." That is precisely why retrospectives are a thing - "What went good, what went bad, what can we do to fix the bad?"

If one of these PRs causes a defect to be reported, as soon as you see the bug has been reported, if you can link it to the original ticket then do so. Also make sure that it is assigned to the person who introduced it (only because they have the most recent experience in that area of the code and will be most likely to solve the problem quickly, of course, not because "you broke it, you fix it").

There are many ways that you could address this - some will be more successful than others, and a lot of it depends on your organization's processes, as well as the personality of your team (including your supervisor).

Answer 9

Processes should be designed to get the work done accurately and promptly. If people are routinely circumventing the system and the system is well designed you should be able to cite a number of problems that have been generated by the circumvention. You and/or your boss (maybe your boss armed with data from you) need to make a specific list of these problems-that carries a lot more weight than just complaining about the circumvention. If circumvention is as common as you say and you can't make such a list, the processes are wrong. They are in fact getting in the way of getting good work done. It is time for a careful review of the process. The evaded review steps are not creating problems, so get rid of them. See what problems would have been caught by what reviews. The organization then needs to define what reviews are mandatory, enforce those norms, and make the reviews a priority so they happen in a timely manner so they don't slow down work too much.

Answer 10

Your input is useless if not written down. Therefore I'd propose to set up a logging system, which logs all actions, performed on a certain task:

Once somebody has implemented something, the commit hash is added to the bug report, and every additional task (code review, unit testing, ...) also gets added to the bug report, in such a way that you can easily find out the following questions:

• Which percent of the bug reports actually passed code review?
• Which percent of the bug reports actually passed the whole development chain?
• ...

Also, it must be possible to log why something is not done:

• code review not passed because of business prioritising.
• Unit testing incomplete (only 20% of the tests are done)
• ...

Without such a logging, you're just shouting in the dark.

Answer 11

You are right. Everyone else in this situation is wrong.

It sounds like you need to continue being "that guy" that annoys everyone by insisting on the process. Your boss isn't taking leadership on this, so you must do this instead. Pushing directly to master means that it is only a matter of time before your product will have a quality escape that impacts your customers and impacts your team.

You want to be the person that says "I told you so" in this case and has the communication (emails, etc.) to back this up. This should put you in a better position--you may even end up with your boss's job.

Another thing to consider is asking for better tools that make it easier for people to follow the process and harder to force push to master. GitHub and GitLab have a protected branch feature that only allows project owners to push to master. You can even lock down your repository so that merge requests must be approved by another developer and a QA person before they get merged. You can also get a build server that runs unit tests automatically on a merge/pull request. It sounds like your boss is on-board with this even though, so it shouldn't be too hard to convince him to start using for better tools.

Do not just wait for things to change after somebody notices a big screw up. You don't have control over what happens if management notices the development team is making big mistakes. Call out the problems early and often for your own sake as much as the rest of the team's.

Of course, if you're tired of fighting, you always have the option to leave, but this could be a career-advancing opportunity for you if you choose to stay.

Answer 12

I had the pleasure of attending an Agile class given by Bob Martin ("Uncle Bob"). If you don't know him, he's one of the founders of what we call Agile.

The purpose of Agile is to get "that feature the customer wants to see on Oct 1st" in front of the customer on Oct 1st, OR, make it very clear to your management on say, July 1st that you are never going to finish that feature by Oct 1st. In turn, your management makes it clear to your customer on July 2nd, that they are not going to see that feature on Oct 1st. Unless, some kinds of changes/tradeoffs are agreed to. That's what management is supposed to do.

So despite having all the technical trappings of Agile in place, it's clear to me your company is not really doing the important part. Management needs to know what the customer wants, and when they want it. They need visibility (some trustable quantitative metric) on where the developers are. This information needs to be continually discussed and adjusted with the customer as time goes on. That's Agile.

Code reviews, TDD, pair programming, and refactoring are technical tasks that enable good software quality and craftsmanship. However, these things alone do not mean the company is using an Agile process. Manangers need to manage using data obtained from these processes incorporate customer feedback to adjust timelines as required. It's that simple.

The situation you have is the developers are trying to use good software craftsmanship techniques in a company that is not using an Agile management process. It sounds like chaos, where various people are making various promises in an uncoordinated fashion. As a developer, there is nothing you can do about it.

Answer 13

The other developers view of the situation is that they will probably be here for another year at most so letting the code rot is cheaper than daily arguments about process with various people who dont value careful engineering.

I think that this is the main problem. If devs feel like they are only going to stay with the company for a short period of time then skipping best practice to Get Stuff Done doesn't seem like a big issue.

Why do devs feel like they are only going to stay with the company a "year at most"? That seems like quite a short period for anyone to plan to work for a company.

Answer 14

There are multiple ways to do the organized development, depending on the team and product. The flow that is now typically pushed assumes everybody working on everything and contributing frequent but small changes to the same master branch, but through the code review and pull requests.

This is not the only way to do the organized development. If "processes are not followed" yet the development goes well, maybe some other rules and processes are actually followed: pair programming, code ownership, release branches, feature branches, development branch, test driven development or something the like.

If so, then may be better to discover and capture the actual processes rather than trying to fix that is probably not broken.

Answer 15

What can I do about this?

Your boss has told you that you can push back so I would avoid feeling pressured by ignoring communication that is designed to pressure you while you're already working on an item. This will train others to stop trying to pressure you.

After reading some other comments that dealt with company culture: You can only improve company culture if you're in a gate keeper position (not necessarily management) where you can block something from deploying and your boss will back you up. This implies that the grandboss will back up your boss... that the grandboss's boss will back him up and so on.

I'll acknowledge Gregory Currie's comment that: "There is a reasonably large difference between telling someone something, and convincing someone. Telling someone relies on authority, convincing can be done a number of ways"

I have worked in places where the value of doing things the right way couldn't be shown because management kept allowing unrealistic schedules. I haven't seen it work where people are convinced without an authority backing good processes.

Usually if things are going a certain way it's because that's exactly what management wants, regardless of what they tell you. That guy you work next to who has zero care for quality was hired by someone who knew he was that way or didn't care that he was that way. If you have an unreasonable deadline it's because multiple people above you agreed with that. If you can't imagine why shoddy code is being released, your boss can imagine why and understands why.

Ultimately everything we do as developers (in a company) is driven by business needs. The business side may have a legitimate reason to force you into rushing code out like they know that the company will fail in a short period of time if clients can't be shown new features and waiting for the features to be higher quality would take too long.

I have seen companies where they had the struggle you describe. And then within 1-2 years everyone is laid off. Management knew this was coming long before the developers.

Answer 16

• Your coworkers don't see long-term prospects at the company so they have no interest in following process.
• Your boss is simply paying lip service to process and has no interest in enforcing it.
• The departments dependent on software don't care about defects, just things they can show off to clients yesterday, so they don't care about process either.

This is an extremely common scenario at companies who don't understand that their most important product isn't the commodity or product they sell, but the software behind it. Such companies will never prioritise doing software "right" because they don't see any value in it.

Unless you are in a position of power at such a company, there is nothing you can do to correct this perception. As such, your only option - if you wish to retain your sanity - is to find a job elsewhere, with a company that understands that quality software is the bedrock of their success, and indeed existence, going forward.

Answer 17

I concur with the other answers that the process should be in place for a reason and followed.

I also agree that it is your boss's job to fight this fight with the business stakeholders and it should be down to them to explain that releasing at break neck speed drastically increases the risk of show stopping (business ending?) issues being released to live.

That said, one way to put an end to this skipping of the process is to implement a technical fix. You can 'protect' the master branch and disable the ability for people to push to it without an appropriate review process:

https://docs.github.com/en/github/administering-a-repository/defining-the-mergeability-of-pull-requests

This can also be done in some other repository management solutions such as TFS.

If you take away the ability for the developers to rush code to production, then the pressure on them to do so is also removed. This moves the pressure up the chain to your boss (where it should be) and it is then down to them to have these arguments.

Answer 18

To start with - this is not your responsibility to fix this, but it is nevertheless a good thing to be process oriented.

You may want to suggest to use some CI/CD system which will only allow deployment when all criteria (tests, ...) are met. Otherwise the pipeline fails and the change is rejected.

If you have a tight enough control of the pipe, then such shortcuts will fail. I would also clearly document the exception process so that there is something to point to when the salesman or whoever requires an urgent one.

Answer 19

Talk to your boss again. Your boss must state this is law from now on. If he doesn’t want constant fights, make up sufficiently hard rules for exceptions so that people don’t take them lightheartedly.

If this goes on for too long, people will get accustomed to it, and it will be harder and harder to change. Maybe the boss has to involve his boss.

You alone can't do much without backing. You could try to persuade your coworkers that following the process is good for their CV, and the ability to politely tell no will further their career anywhere. This is true, and also hard to sell.

Answer 20

One crummy thing to consider, in response to this quote from the question.

My team has a variety of development processes that code is technically supposed to go through to get to the master branch. Things like unit testing and code review.

No - you don't. The process that happens is the process that you have and the process that the team, the entire team (from managers on down), actually values.

If there is a document somewhere or a nebulous set of ideals in a few developer's heads, that's fine, but it isn't your process. One thing you can do is to either become comfortable with your actual process, realize it isn't ideal and live with it (and communicate the consequences) or convince the development team to implement structural changes that more tangibly enforce the processes you want, such as: merging cannot physically happen outside of an approved pull request, automated build pipelines, etc...

Good luck - that's a crappy situation to be in as a developer

Answer 21

I'm not an expert in this, but my 2¢ would be this:

Each time tests/process are pushed back, make an estimate of the number of bugs that this equates to, plus, the damage it equates to in terms of loss of money for the company, plus the work-hours that will be required if this becomes a legacy fix (which is typically much larger than the amount of time taken to follow test-driven development in the first place). Unfortunately, this obviously requires a bit of homework on your part, which is probably beyond your job description, but at the same time, rough calculations are fine, and you can probably get some sort of idea for this from previous bug reports that related to missed tests, etc.

Make sure to stick to these figures, and update them each time tests are skipped. Then, at the end of each meeting, in a business-as-usual (but not passive-aggressive) tone, make a casual note of "accrued technical debt so far", as per those numbers. People will get chuckle at first, then maybe get tired of hearing it, but once this number starts climbing, it might cause people to take notice. At some point, hopefully, it will reach a tipping point, and the discussion might go like this:

Boss: Update me on yesterday's push to production.

You: Certainly. Yesterday we pushed 5000 lines of code to git. Due to urgency, we skipped tests as per request, estimated at about 30 unit-tests for this commit. From previous experience, we have found that 1 in 3 skipped unit tests manifests itself as a user bug-report 2-3 months down the line, with an estimated cost of roughly $10,000 in sales, and 40 person-hours of legacy fixes per bug, so yesterday's commit costs an estimated $100,000 in losses and 400PH technical debt. Given our previous technical debt estimate of 470 bugs, minus 30 specifically missing-test-related bugs that we fixed over the last month (we spent an estimated 1200PH doing this), plus today's estimated technical debt of 10 bugs, this brings our accrued technical debt today up to an estimated 450 bugs, which, at an estimate of $10,000 / 40PH per bug, this leads to an estimated $4,500,000 loss for the company plus 18,000PH worth of technical debt so far.

Boss: ... Wtf. Ok, fine, let's think about this. Pushing early without tests has generated an extra $X for us than if we had pushed late ... but if what you're saying about tests is true, it has also cost us $Y. Maybe we should put some thought into whether the technical-debt cost $Y actually offsets the early-shipping profit $X in this case ... how much extra time did you need to enforce those tests anyway?