59

As some background, I am the Team Lead.


There is a Senior Developer in my team who does not push his code at all.

He commits to the master branch directly and puts code into a production branch without it being properly versioned first.

He does not open feature branches according to the workflow, and when asked to organize everything, he comes back and says that he had higher priorities like deploying new stuff into production.

From my perspective, following some simple practices in a development workflow, and having all code properly controlled and uploaded to the Git server is part of any task that involves dealing with compiled code or artifacts.

For me, the code in our master branch should be deployable and development branches should be reliable. This is to make it possible for other developers to open branches.

My final words on this topic are: Having the code properly versioned, and merged into proper branches, is "sine qua non" for any development task. Any senior developer should take good care of code.

How to convince this developer to stop doing this?

What are the risks from him continuing this?

Is this a blocker for other team members, or am I wrong?

  • Comments are not for extended discussion; this conversation has been moved to chat. – Jane S Nov 8 '16 at 11:52
  • 10
    I don't have a lot of patience for people who willfully won't take instruction. Give this person one chance, emphasizing that adherence to standards is a job requirement. If they continue to fail to comply, fire them. People like this are a huge efficiency drag. – Brad Thomas Nov 8 '16 at 14:13
  • 20
    Github, Gitlab and most git servers allow for protected branches. Lock your master branch so no one can push directly. – spuder Nov 8 '16 at 14:14
  • You bolded the "senior" word to mark the term as ironic in this case, right ? I hope so... – Radu Murzea Nov 8 '16 at 14:59
  • 1
    @Testa It's been a while since you asked this. Are there any updates about it ? Was this issue resolved ? If yes, how ? – Radu Murzea Dec 1 '16 at 21:12

10 Answers 10

114

The other answers cover a lot, but I will add something slightly different.

Why do your processes allow him access to any production server in an open manner?

You should be setting up your deployment system so it is automated, reproducible and closed - no one should be able to circumvent the deployment system.

  1. Remove all developers access to the production servers. Allow them remote access to the system logs and the debuggers, but that's it.

  2. Run all your code build and packaging via a Continuous Integration system, such as TeamCity.

  3. TeamCity builds your specific branches and tags in git - have it run the unit tests and integration tests.

  4. Deploy all your code via an automated deployment system such as Octopus Deploy - this will take the packaged output of your CI system and deploy it in an automated and reproducible fashion into any environment you have set up. It can deploy specific branches to your dev environment, promote those packages to UAT or Production, and it will never allow your developers to muck around in production. It also gives you a full audit log of whats been deployed where.

Use the processes to enforce the restrictions on your developers - until you do, you will always get someone circumventing your manual processes because they are "better than you".

If your developers don't have admin rights to the production servers, they can't circumvent your processes.

Your deployment processes are part of your system - in the event of a server rebuild, you should be able to deploy all your code in an identical fashion as before. You will find this incredibly difficult to do when the deployment process is manual, as so much is undocumented, while if the deployment process is automated then at least the automation configuration acts as minimal documentation - you just click one button and have all relevant packages deployed to the new server. No fuss, no problems, just done.

Edit: Also, see my answer to How to avoid bad practices of work by employees, as it is also relevant to your situation. You need to act as the gatekeeper, and to do that you need to actually pit some gates into practice.

Second edit: a few people in the comments have expressed the concern that this isn't an immediate fix - unfortunately, unless the developer in question has an overnight epiphany then nothing short of that developer leaving will be an immediate fix.

TeamCity and Octopus Deploy (you don't have to use those, there are others out there) are quite easy to get up and running - they should take a day or so to get installed and configured, plus an hour or so of per project configuration.

However, even just starting the process of automating the deployment process should trigger something in the problem developer - it should immediately show them which way the wind is blowing, and that may be enough to have them make a decision one way or the other. Should they stay and fall in line with the team lead, or should they continue to be insular and risk having to leave...

One other thing to consider is that what this developer is doing is showing other team members that they can get away with bad practice - I'm betting there are other team members who quietly do fixes in production, cover up mistakes etc. These are the sorts of things that will quickly lead to you not being able to rebuild an identical production system if that need should arise. And believe me, it will arise.

  • 59
    And if the maverick developer decides to quit over losing production access, you are probably better off. – Jim Garrison Nov 7 '16 at 0:25
  • 1
    @jpmc26 I agree with all of that. I did upvote this answer for contributing real, useful and even technical information to the problem. In general good companies are better at not having problems then in dealing with employees who do have them. Again this isn't the answer to the problem, but it's very good, relevant information. – user42272 Nov 7 '16 at 6:51
  • 7
    This answer is suggesting a huge change to processes and technology, without actually addressing the real problem, which is that the employee is flagrantly refusing to follow simple and clear directions from his supervisor. Persistently. Over the long term. Sack him. – Lightness Races with Monica Nov 7 '16 at 23:32
  • 2
    If it's a small team (two or three developers who each wear many hats), sometimes you don't have a choice but to give them all production access. But with such a small team, each member should trust each other 100% and there should be processes in place that prevent this behaviour from occurring. It seems like this 'senior' developer is not trusted. That's the biggest problem now. – user56728 Nov 8 '16 at 3:19
  • 6
    @LightnessRacesinOrbit trying to solve personal and cultural problems with processes and procedures should be the first step. Not only will address this specific case but it will avoid repeating it in the future. – angarg12 Nov 8 '16 at 10:14
35

Let him know that he is engaged in insubordination, and there will be consequences to his insubordination. Tell him that part of your job is protecting the integrity of the software engineering build process, and he is making a hash out of it.

If he continues, cut his access to the the Master and the Production server. Then tell him that he is on a PIP where his first task will be to explain to you how he intends in the future to upload his code to the Master and the Production unit (*).

As team lead, you need a senior developer. You don't need a headache. In particular, a headache who plays prima donna and acts as if the rules are meant to apply to everyone but him.

(*) He has to tell you, because you already explicitly told him, and he didn't take what you said seriously enough to get it. When reason doesn't work, try religion e.g. the "lightning bolt on the road to Damascus" thingie - I am extremely flexible in my management methods and approach :)

  • Comments are not for extended discussion; this conversation has been moved to chat. – Jane S Nov 10 '16 at 23:48
30

And when asked to organize everything, he comes back and says that he had higher priorities, like deploying new stuff in Production.

Of everything in the question, this looks to me like the best angle to act upon. Who is setting his priorities? If your teams are organized like ours are, it's the team lead's responsibility to set his priorities. Thus, let him know that you recognize that you are not communicating your priorities for him sufficiently, and that you will be taking extra effort to make sure they are communicated properly.

An individual like this will most likely have a reply cued up, because this wont be the first time someone will have told them this. I can't say what that reply will be, but it will be a lot closer to what the actual issue is. It will be out in the open so you can deal with it better.

Perhaps I am being too subtle. Just in case, I'll be more direct. Anyone in a "senior developer" position should know just how badly they are messing up when leadership offers to "help" them by "taking extra effort to make sure priorities are clear." They should realize rapidly that what that really means is that they are about to lose all freedom of choice in their approach until they shape up.

  • 7
    @VietnhiPhuvan I phrase it this way intentionally. It is entirely possible that there is a communication issue that is permitting the senior developer to have this strange idea that they set their own priorities. On the other hand, if that communication is indeed working fine and the developer is just an obnoxious individual, then surely they understand what it means when a lead tells them "I will take extra effort to make sure priorities are communicated properly." Anyone who has been around long enough to be a "senior" developer should know what it means when leadership offers "help." – Cort Ammon Nov 7 '16 at 1:51
  • 4
    Kind words with a hint of steel - I like that :) I haven't used that style of communication in a while. In fact, I haven't been subtle about things for a long time :) The possibility you have raised is the reason why I am no longer subtle about much of anything. +1 for clarifying your position. – Vietnhi Phuvan Nov 7 '16 at 2:09
9

Make it clear that this is absolutely essential.

Have you done so already? As a manager, I know it can be hard to confront employees directly when they are doing something unacceptable. You mention there was a conversation something like this:

You: Can you please organize your code and properly use the development workflow we discussed.

Senior dev: I haven't gotten to that yet because I need to get these urgent features out.

What I'm wondering is, what was your reply to this? Did you make it clear that you disagree with this prioritization, and from now on, no new development should happen until he implements the correct process? If not, then the senior dev really hasn't been given a clear message. He might be left thinking his choice was the right one under the circumstances.

Step one is to clearly state what you expect to happen.

I see a lot of answers suggesting that you let him know that his job is on the line, etc. But don't do that until this basic step has occurred. He needs to know that this development process is mandatory, and getting it set up takes priority over everything else.

Step two is to provide support for him to learn the process.

I would offer to him to have another developer show how it is set up. You say that his lack of Git knowledge is the problem (despite claiming to know). Unless you are 100% sure of that, there is no need to address it directly. Simply point out that it would be easier and quicker for him to see how someone else already has it set up, and it would ensure that everyone is doing things the same way.

In the long run, if there is a consistent pattern of hiding the fact that he doesn't know something with workarounds, then this is an issue that needs to be addressed in some way (as it will be quite detrimental).

Step three is escalate - only if the above don't work.

Other answers have given pointers on how to do that.

Making your preferred commit path mandatory is a good practice on its own merits, but don't use it to solve this problem.

The basic issue is the senior dev needs to respect your role as team lead, and accept practices that you put in place. (Also, conversely, you need to clearly communicate your expectations if this isn't happening). Not doing this, and instead papering over the problem with technology, would be a mistake.

However, once the above communication has taken place, also making the process mandatory would be a good idea.

  • I agree that a technically forced solution is not necessary. Most of the dev environments I have worked in allow developer access to master branch, and developers often have secondary role as ops and/or support with enough access to patch production servers. In these environments the development process is followed closely and works fine thanks to team members understanding and respecting the process, and having enough self-discipline. Technical fixes with restricted rights are maybe better fit for larger teams (with multiple installations to protect and full sysops team to manage them) – Neil Slater Nov 7 '16 at 10:17
  • 2
    @NeilSlater the technical solution can be helpful as a way of helping the team be disciplined to good practice, and avoid dumb mistakes. But it shouldn't be used as a substitute for management. – user45590 Nov 7 '16 at 10:19
4

You need proper code control and review. If he understands the process and violates it without excetionally good reason, you don't need him on the team. Tell him so.

Priorities: it is your responsibility to make the priorities clear. If he goes off in a different direction after you have done so and he has acknowledged, you don't need him on the team. Tell him so.

Remind him, if necessary, that the team lead's evaluation of his work affects his year-end review.

Give him a fair chance to explain why this one case was special... but if he doesn't have an adequate defense (in your judgement, not his) and he isn't willing to promise it won't happen again, it's time to have a talk with his manager.

  • Would you guys have examples of people who lost their Job for making mistakes like this one? – Vampire Nov 6 '16 at 23:28
  • 1
    If he doesn't follow company/department policy, he can be fired for cause, especially when he endangers the project by doing so. Examples are not useful, would violate privacy rules, and are frankly irrelevant. – keshlam Nov 6 '16 at 23:33
  • 1
    This answer is useless because it doesn't address properly educating the employee beyond "fire him if you don't educate him." – user42272 Nov 7 '16 at 3:48
  • What do you talk with his manager about? Maybe do that first then talk to the employee? – user42272 Nov 7 '16 at 4:10
  • @gusdor: Or at least moved to a job they are able and willing to do. – keshlam Nov 7 '16 at 12:39
2

There are 2 lines of attack necessary here:

  1. You need a process that makes it easy to do the right thing and hard to do the wrong thing.
  2. You need the team to agree/accept what the right thing is.

The answer by moo already covers 1), so I'll focus on 2).

Your team should know that they need a stable branch, a somewhat stable shared branch, and a common merge strategy that ensures people know which change is in what branch, which means everybody merges things through branches in the same order. If they don't know that, send them to mandatory training that's held by a third party. Yes, it's expensive, but having a team that doesn't understand branching is more expensive in the long run.

You can usually leave other decisions, such as opening a temporary sub branch for some feature, to the team as long as you must ensure that decisions that affect the team are taken by the team, not by a lone developer.

You can facilitate the teamwork with retrospectives, meetings where after each feature/milestone/sprint the team gets together and discusses how they could have done better, as a team - the frequency of these should be about once every 1-4 weeks and you can find tons of material about retrospectives online. If the team knows about branching, during the retrospectives your lone developer should get flak from the entire team, not from you. You can then guide the team to put automated processes in place which make it easier to do things right, and very hard to do things wrong. These are the processes described by moo's answer.

Generally speaking, as a direct supervisor, you can't simply implement and enforce a large process your team disagrees with, without a very high risk of fallout. If they don't know about proper branching, you need to train them on "the right way", then you can implement and enforce minor processes to force them to work as a team: One common way in your case is to demand a code review on a merge to Develop and Master; the next time a merge has been done wrong, you can now ask 2 people what went wrong, the coder and the reviewer.

  • My first guess was the developer was not confident with Git's branching. – GameDeveloper Nov 8 '16 at 11:51
1

You do this by instituting hard, physical policies that require that changes be checked in a specific repo or they won't get deployed, which means work doesn't get done, which means he doesn't do his job and that is grounds for termination. If you're team structure is "laid back" and developers have keys to production and can deploy on their own, or have DB credentials to deploy DB schema changes themselves and not going through the proper PROTOCOLS ESTABLISHED BY MANAGEMENT, the conclusion is simple: you are not managing well. Managing, in a software development environment, is mostly designing the team rules and protocols by which team members MUST abide, not babysitting developers on a daily basis.

You do not beseech your subordinates to do things a certain way. You have all the tools necessary at your disposal to force them to follow proper protocols. This is not a problem with your employee but a problem with your organization and a lack of structure and rules that make life simpler, easier, and more efficient.

  • This reads as if management is the job of making people loathe their jobs by putting protocol in their way. I'm so glad I don't work for you! – reinierpost May 29 '17 at 15:50
  • Following simple protocols is critical to collaboration, I wouldn't hire you if you were unable to do it – amphibient Aug 7 '17 at 14:24
  • I agree protocols are important. But how to arrive at a good use of good protocols? Having non-experts design protocols for experts to follow seems dysfunctional to me. – reinierpost Aug 8 '17 at 8:03
  • Who said "non-experts design protocols" ? – amphibient Aug 8 '17 at 14:07
1

There is one possibility that nobody has thought of: That this senior developer just doesn't understand how the workflow should work, and what he has to do to stay within that workflow. That he never learned how to use git properly in the first place.

If he uses git directly then all of the things he needs to do are actually quite complicated, so hopefully you have some tool on top of that. Sourcetree works quite well; I'm sure there are others. Once set up properly (which can be a pain) it makes life a lot easier.

Take one of the developers who does everything right and check with him how this guy's machine is set up. Does he have the tools that everyone else has? If not, set them up. Can he create a branch? (On my machine that's selecting the branch you want to branch from, selecting one menu item, and typing in the name of the new branch). Without the right tools, it's a pain. Maybe he really doesn't know how to do that.

Go with him through all the things he needs to do. Make him write down all the steps he needs to do. That's what I did the first time I used git: I wrote everything down. The second and third time I refined what I wrote down because it didn't quite work and there were situations that I didn't anticipate. The fourth to the tenth time I used my notes to do things, and then I didn't need them anymore. I could see someone not knowing what tools to use and how to use them, being too proud or obstinate to ask anyone from help, and creating an unholy mess on the way.

-2

Wow. Just wow. This is so deeply unprofessional, it's unbelievable. But talk to HR first to make sure they are on board and that you don't do anything that could be a legal problem for your company. Then you talk to him, taking into account any advice from HR, and tell him that if he repeats this ever again, you will take steps to remove him from the team. Which likely means from the company.

Make sure that HR understands what damage this irresponsible behaviour can cause.

How to convince him to stop: At least two answers here end up with him not being employed where he is now if he doesn't change. That should be convincing. In addition, people saying this is deeply unprofessional and dangerous.

What are the risk: A broken production server is the smaller risk. A production server that makes costly mistakes is the big risk. Imagine an e-commerce server that charges VAT instead of adding VAT to the price. You'll have many, many very happy customers.

Is this a blocker: Not really. When I make a branch, I fully expect the develop branch to be changed when I want to merge, so no difference there. If he goes past develop branch to master branch, that's a different matter. Whoever is responsible for the master branch is in for a big surprise. On the other hand, if I was responsible for the master branch then anything unauthorised would be removed and then whoever did it would be in trouble.

  • 4
    The developer branch is behind Master, which in our Workflow is a problem. I would see that as a blocker. And he has code in his machine that's nowhere, but compiled and running in Production. In other words, if his machine is stolen, we don't have the source code of what's running. – Vampire Nov 6 '16 at 22:19
  • 4
    While I would call some things unacceptable, production code that is only on a developers machine is beyond unacceptable. "If his machine is stolen" - is it not even backed up? Hard drives and SSD drives die. So the code will be gone not if his drive dies, but when it dies. – gnasher729 Nov 6 '16 at 23:43
  • 6
    It's not really good management to have no idea what to tell your employee besides "If you mess up again I will fire you." – user42272 Nov 7 '16 at 3:52
  • 4
    Too much escalation too soon. It's not even clear that the OP has outright said "you have to do this". – user45590 Nov 7 '16 at 8:01
  • +1 - Can't add more, sorry :-). It is astounding how much people are missing the key point here, which is that the prospect of imminent catastrophe from a single event has been introduced, and people are rabbiting on about PV procedures and proper means of escalation while the OP stands on the edge of an abyss. Quite amazing. – Russell McMahon Nov 9 '16 at 11:30
-2

Summary:

  • Apart from the technical & societal issues raised by others you have a major immediate need - the actions of this single person may be producing the chance of a single point catastrophic failure.

    • ie While most 'disasters' happen because several apparently unrelated and statistically improbable events happened to occur simultaneously. In this case a single event may be enough.
  • While your diligence is hopefully directed at optimising outcomes for your employer, sparing a thought for your own position is also wise.

    • ie The developer, and therefore as his manager you, may be (and probably are) putting the project at risk of a major loss (measured in terms of time, resource, need to replicate work, system integrity, customer confidence and more, and ultimately in $. If such an event occurred and was severe, even if not "worst case", there is no reason to expect that you may not lose your job.
  • The above loss, of as yet unknown magnitude, might occur at any time - next week, today, or never.

  • Systems are normally arranged so the scope/range/magnitude of the consequences of any reasonably forseeable events are knowable. In this situation this is not the case, and this needs remediating immediately.

    • while you know that the employee's actions have placed you at risk, you are so far unaware of the potential consequences of probable, possible and worst case events occurring. Establishing what you do not know is the first step in setting things right.

    • You are in a position to take immediate action to determine what is unknown to you about the whole situation. Do so. Now. Then establish what steps are urgent and which may be slightly delayed and then set about remediation

__________________________________________________

New question for your list:

Q0: "What are my most urgent priorities? "

A0: ENSURE that what you have is stable against catastrophic loss.

It does not sound like it is.
If not, make it so.
Now!

_________________________

The possible short term dangers & consequences & best path forward:

The following is not meant to be rude but is meant to be 'robust'. By all means say if you do not agree. If you do agree. do something apposite NOW.

You (or others) mention "machine stolen, uncertainty re "backups" etc ... .
If you do not KNOW if you are covered against any single point catastrophes then it is YOUR 'fault' if something happens that wipes out an unknown proportion of your system instantaneously.
If that happened in any significant way you should be fired and probably would be, after you had implemented whatever remedial activities were possible.

Without further input, look at your 2nd (now 3rd) question, decide what risks MAY exist that you do not know about because the system is not controlled and take immediate steps to eliminate those risks. This may involve freezing all activity and/or files and/or ??? by the developer. So, involves them in the process.

______________

Suggestions for points to form part of an (urgent) interview woth the developer:

I understand xxx is the case. Is this so yes/no?
If I understand wrongly, please clarify.

OK - if yyy happens what will the outcome be?
(If answer is wrong say why and iterate until agreement reached.
(If iterative loop will not close after an excessive period use '45 Clear' or acceptable local equivalent :-( :-))

Once conclusion as to consequences are reached. This is unacceptable to me. This puts me/company/system in an unacceptably hazardous situation. This MUST be addressed yesterday. I am about to implement xxx. Do you agree? / Can you suggest alternatives? Again, iterative loop may result. Again, use of "GOTO" may be needed ( or not :-} ).

If you do not do this or equivalent you will be fired as a consequence. Sometime.

  • 8
    Honestly, I have no idea what you are proposing here. – user45590 Nov 7 '16 at 11:00
  • This answer has some confusing formatting. I think the risk assessment is the only easily salvageable part, you might want to split that out to a new answer. – user30031 Nov 7 '16 at 14:05
  • @dan1111 Proposal: "ENSURE that what you have is stable against catastrophic loss.". Reason: As gnasher said: " While I would call some things unacceptable, production code that is only on a developers machine is beyond unacceptable. "If his machine is stolen" - is it not even backed up? Hard drives and SSD drives die. So the code will be gone not if his drive dies, but when it dies". || He is presently inviting absolute catastrophe. – Russell McMahon Nov 7 '16 at 14:50
  • This answer is so general, it can be copied and pasted to almost all questions on workplace. Yet is makes no sense if I read as much as the formatting allows to my mind. – phresnel Nov 8 '16 at 7:56
  • @phresnel You may or may not be conversant with what can happen in a major programming project. Most workplace activities do not involve team members taking a well established structure where things work well and converting it through the actions of one person to a system where single point failures may cause massive catastrophic failures with unknown consequences. A programming environment can be like that. Whether that has happened here is unknown to us and probably to the OP. That must be rectified or his job, his customer and perhaps his company may not last a month. – Russell McMahon Nov 8 '16 at 8:52

protected by Jane S May 28 '17 at 1:53

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.