Conflict with unqualified senior colleague from first day of work

Question

I started as a data scientist 3 weeks ago at a small company (<100 employees) and have been assigned the one and only data scientist at the company as my lead/mentor. This person, let's call him "Joe", has been working at the company "from the start" and is liked by everyone including our part-time non-technical manager, our team consists of us 3.

In my opinion, Joe is far from being a data scientist. He started working behind the phone at the company and when they needed to do some analysis, the most suitable candidate was Joe. Since then Joe has been doing analysis and modeling, even though he has never studied these topics and has no prior experience.

I have identified a lot of mistakes Joe made and makes, which leads to providing the company wrong information.

The company is now investing more into data and hired me and this is where the problem starts. Joe thinks that he knows everything and he wants to do everything his way. The manager is also under the same impression. I have therefore been assigned to do what Joe says me to do.

In the beginning, I tried teaching Joe about SQL, Python instead of Excel, statistical methodology, correct validation, etc. But his response is that he does not need it, he already knows everything or he has his own way.

Besides this, Joe also extracts all the data for me, apparently, this is a company policy to protect themselves until they feel comfortable. This makes me very dependent on Joe. When I ask for data, he will extract the data he finds most suitable and not what I want for my analysis.

This week we have had a lot of conflicts due to the project we are working on. I essentially said to Joe this project is going to fail and the result cant be achieved based on what he has provided. He called me into a meeting and said I should be careful and follow his instructions. This got me a bit worried as I have a 12 week trial period.

I liked the company, colleagues and I see a lot of potentials as the company is in an early data phase.

Ideally, I would like to work on my own project without any input at all from Joe. Is there a way I can get this done?

I should probably talk to the manager, but I dont know if I should say that I find Joe totally unqualified. Additionally, I have not really bonded with the manager as we have only spoken twice, Joe is mainly my "manager".

Answer 1

Unfortunately, it seems you have done what you can speaking with Joe. He is so obstinate that you must go over his head to correct it. Tolerating the situation will just bring you more pain and possibly all the blame later. Though I have not been in your situation personally, I suggest:

1. Identify someone over Joe's head with the power to change things (Director, CTO, etc.)

2. Develop a clear ask, e.g., "I want to work independently of Joe, and I want to improve your data operations with the following technologies..." Do not put Joe down here.

3. Provide a clear report of the mistakes Joe has made and the cost of such mistakes if any. Again, do not put Joe down unnecessarily. Do not call him unqualified. Just convey the facts and consequences.

4. Justify your ask. Quantify what you can offer if let to work independently, promote the tools and techniques you want to use.

5. If none of this works, quit as soon as possible to avoid prolonging the pain. You are in a trial period-- that goes both ways after all.

Remember, the company hired you because they are getting more serious about data. Correcting their processes is exactly what they need from you, and is frankly and opportunity for you to lead and gain credibility.

Answer 2

Ideally, I would like to work on my own project without any input at all from Joe. Is there a way I can get this done?

Probably not, but you could ask your manager if you could work on your own project unassisted, to "see how it goes". Avoid overt criticism of Joe - based on what you have written, that wouldn't be well accepted.

Realistically, based on everything you've written about Joe and the company, you should plan on leaving either at the end of your trial period or before. You don't seem to view the colleague or company as a "fit" for your desired job situation.

Start looking for new jobs now. Next time, make sure you know who you will be working with before accepting the offer. Try to arrange part of the interview to involve talking with any senior colleagues. If you ask, most hiring managers would be happy to accommodate such a request.

Answer 3

You’re in a difficult place. Joe has reputation, internal company knowledge, personal relationships built up over years and many years worth of self taught knowledge that may or may not be valid.

You assume it is not, however you don’t need to be formally trained in something, to have inherent expert knowledge of a subject.

Now we come to your difficult place, keep in mind the company is as successful as they are because clearly Joe has been doing something right enough, by outright telling him he’s wrong or unknowledgeable, and also telling others this, they simply won’t believe you. Afterall they’ve always had Joe, and things have been working great.

With that said, that doesn’t mean Joe actually know’s what he’s doing. All of his success could be based on pure enough luck and customers who know even less than him.

Here comes your formal knowledge. You’ve learnt all the “right” ways to do something (the right way is the way that gives a usable result, it doesn’t have to be python and sql)

And you know how to apply it and how to use it in a scientific way. Which good on you. But well you just don’t have the status and you don’t have a college who is willing to listen to a formally trained person in the subject who they themselves also feel like an expert in.

There is a great saying, something along the lines of, “the ignorant are too ignorant to know what they don’t know”

So you will never convince Joe he doesn’t know better, and you will never convince his colleagues that he doesn’t know better, their anecdotal experience says otherwise.

You could simply go business and state to the people who care about money and flat out say, “you hired me as an expert, and i’m telling you, this this and this is wrong and you’re selling bad data that will ruin your PR in the future when the right company realizes they were sold bullshit”

You offer them at the same time and fully thought out description of what is wrong and why, and then you offer your solution. They now have the choice to deal with it or not. An ultimatum is rarely a good way to go with anything and as awesome as the company is, you’ve already lost the game with Joe. You went about it wrong to directly confront his methods as an attack, and he will forever be against you, even if you can convince the company to support you in this project, Joe will always have the reputation behind him and his colleagues support to make your life terrible there.

Either you decide right now to just deal with his shit and spend the time to be the best you can with the garbage you get and build up your own reputation and good results, or you give them the ultimatum and then suffer the consequences later.

Either way you lost.

Answer 4

It sounds like Joe isn't a great manager, but unfortunately it sounds like he's too arrogant to change.

Personally, I'd be looking for a job elsewhere and have a plan for that. When you're ready you can then go above Joe's head and have a frank conversation.

I wouldn't go in with "Joe is unqualified and I don't want to work under him", by a bit softer with your approach. Something like:

I've tried to bring this up with Joe already, but I'm concerned about some of the methods being used and some of the numbers we are sharing with clients. I have identified ways we can improve our method to reflect our work more reliably.

A manager doesn't necessarily need to know more than you, but if that's the case they need to at least understand that. They've employed you, a qualified data scientist, for a reason and hopefully they'll spot that they need to sit Joe down and ask him to listen, but there is every chance you've joined a company where they all think the person there longest knows best. It's better to know that now and get out if that's the case.

---

The other option is to wait out your probation, then the extra year/two years until you've got everyone's trust, but the negative to that is if someone spots the mistakes and you've been doing it Joe's way you may (as the newest person) get thrown under the bus anyway. Plus I suspect you'll be unhappy.

---

A tip for next time: get to know the situation you'll be working in before you accept a job. As I mentioned, there is nothing wrong with your manager knowing less than you about the area you'll be entering, as long as they accept that. When you meet them, ask lots of questions, it will become immediately clear how much they know but also if they are aware or not.

Good luck!

Answer 5

I think the way you need to go about this depends on your hierarchical role with Joe.

If he's your manager, supervisor, etc, then really you need to just suck it up and ultimately do the things he asks.

He can be wrong, and you can simultaneously be handling it poorly at the same time...

What I mean by this, which other people have hinted at, is it does not really matter if he's not qualified. Addressing this in any manner that insults or degrades or insinuates something negative to his character will make you look bad;

The best thing you can do, is do your best effort to succeed within the guidelines you've been given. And also, don't stress so much on this - It doesn't matter if your project fails. It's okay for things to not work if you do the things you were asked. Document everything you can, put in the work, and try to care less about the results... This is why older, senior employees seem 'lazy' - this experience right here. They stop sweating this kind of thing...

It is very common that you're not able to use the software and tools that you're familiar with at a role. Some people may filter these important things in the interview and screening process before taking a job.

But a good employee will adapt and learn other technologies. I would agree Excel is inferior in ways. But MANY organizations use microsoft products like Excel and even MS Access, with lots of custom VBA script in them.

Suggesting they change, purely because you believe / were tought that another software platform is 'the correct' way to be doing it, will certainly do little more than make you look naive, and unqualified yourself...

Because companies don't care about 'the correct' way it should be done. If for some reason it was built out a certain way, they're hoping a good candidate can further that work and complete work objectives.

Now, don't take all of this as me saying you're wrong - If you know of a good way to improve, I absolutely encourage you to bring it up to Joe / the relevant manager. But you do this lightly as a suggestion, with a brief exerpt of what and why this will improve, and if not accepted, you keep going and do your best to get them what they're asking for within their guidelines...

If they're giving you a request that ultimately will lead to bad data.... just note it down and be prepared to explain why if it fails. Bring up your concerns - LIGHTLY AND POLITELY - if someone asks you to do something you can't do, but if they don't seem interested, you have to let it go. You may be able to have a conversation with your manager about it at some point, but I wouldn't raise this as a personal conflict between you and this other person.

This person / people above you may actually be fully aware that what they're doing is providing incorrect data... it may be intentional.

With all of that said - If Joe is just some extra person, with no actual hierarchical loop above you... I would recommend asking your manager for access to the data source, and explain that you believe you could be more productive if you were able to connect to it directly, you'd save some of joe's time, and you might be able to pursue your own tools where you could get better results from the data.

Maybe a line about, I believe I can write up some SQL queries, and even add some indexes to improve performance in the database, etc, but Joe isn't that familiar with this, could I spend some time working on this directly?

Answer 6

The thing is, the company has given you the work they need you to do, which is what Joe is working on. By suggesting that you work on something apart from Joe, you are also suggesting either that the company's priorities are wrong, or that the company pay you to work on something that is not of critical importance when they have things of critical importance that need doing (i.e. what they've assigned Joe to do, and what they've assigned you to help Joe with). Neither of these are particularly productive; essentially they said "we need to analyze X data" and you're saying "I want to analyze Y data instead". No, the company needs X data analyzed, if they needed Y then they would have asked for Y, but they didn't.

This means, when you ask them for a different assignment, they're (probably) going to ask why. At which point you can address the situation directly, implying (or stating outright) that you believe Joe to be incompetent, or you can make up a roundabout reason to get around having to badmouth Joe in front of a boss who obviously likes him. In the latter case, I find it doubtful that you'll be able to construct a compelling reason to get out of the assignment that will be verifiable (in that it provides the company value above not doing it) but untrue (in that the true reason is that you want to get away from Joe).

What this means is it is doubtful that you can ask for what you want without taking on the issue of Joe directly. Now, this situation isn't as bad as it sounds, because you do actually have leverage. The leverage you have is that Joe doesn't know what he's doing and you can (presumably) prove this to management if you have to. Essentially, you pull the "expert" card: "you hired me to this company because you believe that Joe is not an expert in this technology to the extent that I am, because I have received formal training and he hasn't. So, you can either believe me that Joe doesn't know what he's doing and you can give me what I need to retrain Joe in proper technologies and procedures to streamline our workflow, or you can choose to not believe me and believe that Joe is the expert, in which case I don't know why I'm working at this company and perhaps you shouldn't have hired me, as you already have the person you need" (you may want to rephrase some or all of this when you actually say it to management, but this is the overall message you should send). This solves your problem. If they choose your path, then you go back to Joe with the backing of management and say that management has instructed you to streamline the workflows and upgrade to better/more modern technologies and if Joe complains then you can refer him up the chain. Conversely, if they choose Joe's path, then not only should you quit this company, but you should feel good about jumping off a sinking ship from a company who values ameliorating long-term employees using bad techniques, over using better techniques to provide the best business value.