How to deal with a coworker that writes software to give him job security instead of solving problems?

Question

I have a coworker that primarily develops programs for internal use in the company. They design their programs in such a way that they progressively consolidate their position within the company so that they are gradually more difficult to replace. Some examples:

• Don't check their code into company version control, only distribute compiled binaries.
• Design their programs using client-server architecture so that the programs they distribute are thin clients that send requests to a server they run on their machine; nobody knows how this server works or what it's doing (other than a high level description).
• Whenever anything related to their programs breaks, the only person who can fix it is themselves, everyone else doesn't have access to his code and lacks required knowledge to replicate the functionality of his server.
• Nobody has the time to write a parallel set of programs or reverse engineer the secret server, so we're stuck with what we get from that person.

Since they have developed a huge chunk of programs we use internally, they cannot be replaced, and since they won't be replaced, we can't get out of this situation. And we're becoming more dependent on that person, since they keep designing their code to strengthen their position in the company.

How to break out of this vicious cycle? How to approach management about this?

Answer 1

You need to compile a report for management.

Write a short document outlining exactly why the current approach is leading the company down a dangerous path (getting hit by a bus scenario, for example). Outline security risks, maybe even cite cautionary tales from within your industry, with references to articles, etc.

Also include a list of ways in which this guy's approach is impairing your own work, as well as the work of your coworkers.

Last but not least, make sure to list of recommendations to be implemented immediately, such as adding the code to version control for all to see, and running the server on a VM which everyone has access to. Outline how these measures should in no way impact this person's work, and will simply add security and transparency to the whole process - make it clear that there are no reasonable objections to these measures.

Perhaps sit down with your boss when you hand him this report, and verbally deliver the exact fears you've written here: that this guy is building himself an empire in the company's infrastructure, and that, ultimately, he is potentially dangerous. If your bosses feel that this person may become unreasonable, then you may wish to follow @BillLeeper 's advice and seize control of his machine so that he will be unable to harm your organization. This will, of course, be for them to decide.

Answer 2

This is a management problem.

Before critical code is deployed, it should be version controlled, code reviewed, and at least use should be documented. If security is concerned, pick the right reviewers, and protect the repo and docs. There is no reason why this can't be started immediately.

There is a bigger problem than job security.

Any one of these developers could put malicious code in the company, either by mistake or for their own reasons. At worst, they could actively commit nefarious acts using their self created situation (extortion, sabotage, industrial espionage, etc.). At best, their opaqueness exposes everyone to security concerns, and always leaves a question mark over any audits or accountability. If something goes wrong, who is to say they weren't somehow involved?

Answer 3

Unfortunately, you haven't really told if anyone has spoken about these concerns with the coworker or management. Is it really malicious? Or is your colleague just blind? Or maybe the management is blind?

I have been "that" guy myself.

At my previous job I sometimes had various side-tasks to "make this small tool" or to make something simple. It turns out, there is never resources for internal software... It usually went something like this:

-- Can someone look at the solutions I chose to tell if it's appropriate? -- Come on, we just need a simple tool that simply does this simple operation, do it and it will be fine.

-- Can I create a virtual server on our server for this thing? -- Man, it's just for internal use. Just put it together with other stuff on that broken physical box. Or put it on that box which does we-dunno-what functions. Or put it on your own workstation.

Of course, there were never time resources to write docs. Unless I chose to do it in my free time. Of course, everything I could say when some tools had problems was "working on it".

And then I decided to quit. That was the first moment that someone around me realized that the "small internal" tools were actually important and "simple" stuff is not that simple. I spent couple of weekends writing docs to screw over my colleagues less. Almost a year has passed and I still receive multiple calls every month about how to do something with my internal tools.

Edit

Some comments have pointed out that I should not help them for free. This is generally correct. I wanted to clarify that I am not putting hours of my time into solving their problems, I am just spending a minute to answer a question. Technically it is still true that I am thus enabling and encouraging the existing practices By the way the company has offered me part-time or hourly-pay position to solve problems like I did before and I refused it.

The thing is I am not willing to force my ex-colleagues to "research better" instead of simply asking me "On which machine is the Veeam running?" if I can simply tell the name or the IP address without thinking or say "It should be written in [..]". Besides 2 minute phone call with ex-colleague is usually as positive and relaxing distraction as visiting stackexchange.

Edit end

So what can I suggest? Your colleague seems quite capable, doesn't he? Discuss this with management. Don't tell "he is becoming irreplaceable". Just ask them - what if he leaves? What if he falls ill for a prolonged time? Convince them that the problem is real. They should discuss it with that guy themselves to find solutions. Maybe he just lacks resources? Maybe he needs another person on "internal software" team to make it all nice and pretty?

Answer 4

Most of these answers are WAY OVERBOARD on assuming malicious intent on the part of the developer in question.

Before making a surreptitious image of the server and then perp-walking the guy out of the office, why not just take a breath and try to understand what's going on?

It could very well be that the person in question is over-worked, doesn't have enough resources and would be more than willing to share knowledge. Or maybe he's been doing it this way for a long time and has never received an indication he needs to do otherwise. At a minimum, especially if his stuff WORKS, he deserves a chance to resolve concerns and collaborate with co-workers.

I see no evidence that any of this was attempted in the OP's question. Before considering draconian options, try communication first. If the person had no intent to do harm, you can expect cooperation from him.

Answer 5

There's something that I haven't seen in the other answers yet:

Casually start looking for a new job

Of course, this is based on the assumption that you have already talked with your manager about this. Other answers have provided the reasons why this isn't your problem but that of your manager and they have also given pointers on how to approach this conversation with your manager.

Now, I am looking at the situation where you have talked about this with your manager and then after a reasonable amount of time has passed, nothing has happened about this. You are getting the feeling that your manager doesn't consider this as much of a problem as you know it is.

That's where you might want to start looking for a new job. No matter whether your manager just doesn't think this is a problem or that he simply does not understand it well enough to see the problem, there is something wrong here. (And I'm not talking about the "private" code, but the problem of the manager not doing something about that.)

Such a problem is something you are not likely to be able to change from the position of a developer. However, there are other companies and they do not have the same problems, so you might want to look for a different employer.

Looking at it from the positive side, though, there isn't too much pressure on you right now. You do have a job and you're not expecting to lose that job. You will not have to compromise in order to be able to keep paying rent/mortgage/living costs. You can just casually start looking around and not quit your current job until you find that job that you really like.

Answer 6

All the current answers and most of the current comments only state the current situation or provide suggestions to take extreme steps.

Just to summarize: There are two possible situations: The coworkers are doing this intentionally, in this case they are malicious in one way or the other, and then extreme caution is necessary. Or the coworkers just don't see the potential and actual problems and dangers, they are causing, then they are "friendly" but should be tought to do better.

So, the following roadmap tries two things at the same time: 1) Try to minimize the potential damage, those coworkers can do, if they are malicious, and 2) try to keep them in the company (so they can develop to being cooperative coworkers in future) if they are friendly:

(btw: I know, you are not the boss, but with the information, others have provided, I guess you will have everything in your hands to convince your boss, to take this thread very serious, so this road map addresses what you boss could do, not what you would do. The only thing you can do is draw attention to your boss. btw2: If you boss still doesn't listen, search for a new job and quit as soon as you found a new one. Because that coworkers are ticking time bombs, regardless of whether they are friendly or malicious - that doesn't matter at all).

1.) Silently make backups of everything you can access. Do not shut down systems in the process, shutting down systems could potentially trigger some sorts of booby traps.

2.) Construct a reason, that the working stations need to shut down. If you need an idea, contact me privately.

3.) Extract the hard drives, make a full image, put them back in. Do this over a weekend or so

4.) If the systems have BIOS level intrusion detection stuff, and you can't circumvent those, construct another reason, why those intrusion detection systems fired.

Those coworkers are creating tools for internal stuff, right? So they don't need access to customer systems and the like?

5.) If they have access to systems, they don't need, change passwords, make sure, there is no sort of public key login, check ports for processes allowing non-standard login. Check cron/at jobs, check inetd, check everything running currently. For every single pid, you have to be able to answer, why that process runs at all.

6.) Get some new employee (really new, completely unknown. He must be a really good expert, because he must be able, to take over their job alone for some month if it should be necessary. You can't just take some random graduated student (not even one with highest grade), you need some of those guys, who never visited a university at all but still knows everything) and insert him into that team to support them. Especially since they are causing blockers on the other workers, it can be easily justified. His official job is to support them, his real job is to learn, how they operate.

Step 6 is especially important, because this way, you have a chance, to actually figure out, whether those coworkers are malicious at all.

If the new guy is being integrated well into the team, then you can assume them being friendly, that new guy should be able to implement necessary changes without any need to tell those guys, that there has been any suspicion against them at all.

If the new guy figures out, they are malicious, but they integrate him, then his job is to play along. Learn everything, find it cool what they are doing, and so on. Pay him twice the money, because he has to work twice, because once he comes home, he has to write down everything he learned and send it to some newly formed team who should take over the work as soon as enough knowledge has transferred.

If the malicious guys don't integrate him, then your only chance is to hope, you got enough data backed up (just for the case) and fire that team. Then you may need two or more additional of that super experts I was talking above, to get a new team into that code very fast.

I hope, this road map helps - at least as a source of inspiration on how to handle this. Maybe, in your company, you have some options, that I can not consider, maybe, there are some cultural differences, so you still have to think about this and maybe adjust the plan.

Answer 7

It seems there are some good remedies here to prevent this in the future, but not what to do about it now.

1. Secure the computer. Either have management and an IT expert go over when it's unlocked and un-attended, or go and demand that he unlock the machine and grant access. Then get this monster off the network. Make an immediate image of the HD in case he has a dead mans switch.

2. Fire this individual immediately. Walk him out the door. Don't worry about cause, there will be plenty of evidence on that computer of his. If the company is worried, have their lawyer work his magic, that's what they get paid for.

3. Get the team together. Explain what went down. This individual was acting in a reckless and un-professional manner. He put the company at risk and was terminated for that. It is going to take all the resources we have to get this mess sorted out.

4. Use the team to re-build and re-deploy this work in a proper manner on secured machines etc. The team is going to have to go through app by app and get a handle on things. Don't worry right away about rewriting, just make sure there are no back doors etc., then get the services up on fresh, controlled hardware.

5. Get a security expert in. This guy will probably not go quietly and will try and 'hack' back in to sabotage or otherwise get access to his system. He may also have global passwords to systems he interacted with or obtained individuals passwords over time. IT should trigger a forced password reset on all users and block any outside access for a time (like VPN).

Answer 8

The programmer in question must not be given any new work until the situation is resolved one way or another. All new requirements must go to another developer/team who must follow proper source control and peer review procedures (new hires if need be). The programmer in question can be kept busy fixing defects or "fire-fighting" his existing legacy. Resources must be allocated to reverse engineer the existing legacy and re-implement by appropriate processes. The cost of doing this has to be justified by the existing risk - what will it cost the business if everything this programmer has done is suddenly lost? Or worse, what proprietary (company) data is vulnerable to loss to competition?

It might be worth asking this employee: "what happens to us if you get hit by a bus or decide to take a one-month cruise around the world?" and gauge the response to decide whether he will surrender his code willingly. If co-operative, there's no need for the situation to become adversarial; if there's no sign of concern for the company on his part, time to get busy securing everything he's touched.

Answer 9

This isn't your problem, this is the managers responsibility and role, you're just a coworker and possibly do not have all the necessary information. I'd worry more about my own tasks rather than want to get involved in my coworkers. I cannot see how anything positive would come out of kicking up a fuss about your coworker.

You'll make an enemy of him, you'll show up your manager for being incompetent and give the impression that you have such little work that you have time to launch internal investigations without being asked to or having authority to.

Answer 10

How to approach management about this?

Say that the best practice is not to permit this, for many reasons.

Professional programmers ought to know that this is no way to run a business; and if managers know nothing else they should at least know that (programmers should tell managers and/or managers should tell programmers).

The reasons are hopefully so well-known that I don't need to tell you. They include "backup" i.e. you're in trouble if you lose the programmer (or if they're reassigned to something else), or if the programmer loses their machine.

At least you have "company version control" so you don't need to fight that battle; just make it a job/process requirement that it actually be used.

You probably shouldn't run production software on the developer's machine. A first step might be to insist that:

• Users don't make network connections to the developer's machine
• Software runs on/from a production server
• Software run on the production server must be buildable by someone else (or by a build machine), from source control

Implementing that requires the source code to be checked in, the build instructions published. I'd suggest you do that as a semi-emergency. Allow the developer no write-access to the production server or the build machine (to verify that production code is buildable from version control).

After you've done that (after you know that the source code is in version control and the build instructions are published), then other developers can think about inspecting the source code and helping to maintain it.

Note that Get Rid Of Indispensable Programmer As Quickly As Possible was published by Gerald Weinberg in 1971 (so, really, everyone ought to know this by now). IIRC the original quote is,

"If a programmer is indispensable, get rid of him as quickly as possible."

Answer 11

I don't know if it is just you or others in the company, but everyone can be replaced. There may be delays, but it can and should be done when people don't do their job. Your managers are not doing theirs.

These developers are breaking just about every basic coding standard. The management must have some idea that something is wrong, but they do nothing about it. I don't see them as a solution to your problem.

You need to have job security as well. If there is a specific bug you need to fix, get the source code. If it is on their computer, tell them to copy it elsewhere. If they have rights to production servers, take it away. They can go and complain to management if they want. They'll be doing you a favor and by exposing their incompetence.

Hopefully, someone will realize that all code needs to be centrally located and backed up. This is the company's property and everyone involved should want it secured. Don't let them get away with this mess. They have ownership of nothing and haven't even shown the slightest skill in over-seeing the company's intellectual property.

Answer 12

The question is, how badly do you want to break out of this vicious cycle? Because let's not be cute about this, it's going to cost the firm.

1. The firm will have to spend money to hire someone to write code that the firm controls.
2. The firm has to demand the code from the coder, and back that demand with legal help if necessary. I'll point out that the code was commissioned by the firm, that the coder drew a paycheck from the company while writing the code so the code is the firm's. A failure on the part of the coder to produce the code would at worst be considered stealing, which would be a criminal offense.

Freedom is not free. If the firm is not willing to expend resources to be free of this individual, then all you are doing is flapping your gums. You all are going to have to confront the situation, because if the coder moves away or gets run over by a truck, the firm is SOL.

Answer 13

Consider the reason they are doing this. It's entirely possible that corners are being cut to match time constraints, performance targets, and consistently increasing demands. This often leads to technical debt and one very stressed coder who has no choice but to fix every problem off-the-hoof.

This person may well be writing things in a way that only they can fix because they don't have the time to document, version and maintain code in a timely manner. Trust me when I say this has a thoroughly negative effect on anyone who finds themselves in this position. It's not a cushy role where someone can sit back and relax.

If, as your title suggests, they are not solving problems, there would be no problem. You'd just throw out the coder, with all of their code, because it's useless.

Answer 14

Preventing situations like this is an extremely basic management task. It follows that the management that is aware of the problem is not competent, and the management that is competent is not aware.

Unfortunately, disentangling situations like this is a difficult management task. So since the managers in charge of this developer were not even capable of preventing the situation, don't count on them being able to fix the situation.

The only* way to fix this is to escalate to a higher level of management. If they are interested and able to fix this, you don't even have to explain anything more than you explained to us - just make it less personal by focusing on the programs, and the issues with them, instead of the programmer.

You should know that this is a high risk - low reward option. If you do this, even if it works, the developer and his manager (who is probably also your manager) will suffer, and know that you're responsible. The only benefit you get from doing this is that you're (possibly) following your own code of ethics and honor, but you might lose your job over it. That's why some of the other answers recommend to let it go or to just look for a better job, which is the smart thing to do.

---

*The other way to fix this is to become management yourself, and fix this, but there are quite a few pitfalls involved.

Answer 15

After his own self assessment he has decided both that he does not have a chance to be promoted, and the only reason that the company would have to keep him is that he is withholding code from them.

I don't know whether you agree with this, but if you do, the code could probably be done by someone better. Or if you don't explain why this behavior ensures he can never be promoted.

I think it comes down to whether this situation is worth fixing as much as how to fix it.

Answer 16

This is a task for management. First management should try to discover if this is intentional. If so, a plan should be made to fire the offending people. If it's not intentional, a plan should be made to train the offending people.

Answer 17

They design their programs ... so that they are gradually more difficult to replace.

Not if I was the boss!

There are two problems here:

1. Bad programmer.

2. Incompetent management.

This is, of course, assuming you are representing the situation correctly.

You can't fix problem #1. There is a slight chance that you can address problem #2. This slight chance is if the boss for some reasons simply isn't aware of what is going on. Go to the boss, and tell him of the problems you see and why they are bad for the company. That will likely fail because the boss already knows of the issue and isn't competent to address it, or knows so little about software and managing software engineers that he isn't even competent to understand the issue.

The only real solution is to start by fixing problem #2, which you can at best play a minor role in. The incompetent manager needs to be replaced.

The new manager then has a sit-down with this programmer, has him explain the architecture, and tells him to stop any new development and document all the protocols. In the mean time, he gets a new engineer in that is there to "help" the first engineer with documenting the protocols, putting the software into source control, and making sure the code itself is well documented. This new engineer does any new development, and hopefully bug fixes and minor enhancements of existing software.

The first engineer won't like this. He may quit, throw a tantrum, object noisily, or worse, sabotage things. This is why doing a backup is the first order of business. It would be nice to have a smooth transition from the first to the second engineer, but don't expect that to happen. The plan is to eventually fire the first engineer, if he doesn't quit or turn (even more) destructive against the company first.

You simply can't let this nonsense go on. The longer you do, the more painful it is to eventually fix it. Not fixing it because it will already be painful is absolutely the wrong way to think about this.

In this case I was using the rhetorical "you". To answer the question what you personally can do, start by taking your concerns to the boss, as I said above. Again, that is unlikely to result in anything useful.

The next step depends on things you haven't told us. It can be very dangerous to go over your boss' head. If so, then there is little else you can do other than evaluate whether you really want to continue working there. If this is a small enough company where you can comfortably talk to higher management, then go ahead. It is quite possible that higher management already has a feeling that the low level software manager is incompetent, but of course they're not going to tell you that. This might be the additional info for them to take more decisive action.

Another distant possibility, if your primary objective is to get this mess fixed and you see yourself a long-termer at this place, is to offer to take on some of the internal tool development yourself. That should give you legitimate reason to talk to the first engineer, understand how things work, where the code lives, etc. Eventually, you'd be the tools guy and management can get rid of the first engineer. Then you can ask them to hire someone for that role so that you can transition back to what you want to do. Again, this isn't something I'm seriously suggesting, but is a alternative if you really want to and are willing.