10

I am engineer and in my spare time I pursue a PhD. I developed as a consultant and recently I decided to become a quantitative analyst at an energy trading company.

When I was recruited for this new position I was told that they were expecting nothing but the best, so I felt that I could really develop there.

After getting some assignments and fulfilling those with state of the art tools and methods, I was called over by my manager and another mid range manager to tell me that I should lower the quality (A.K.A use simpler methods. I.e. linear two-variable regression instead of multidimensional deep learning) because "others" might feel threatened by the new developments. They also told me the they still were expecting to be mesmerized by my work.

I did talk to my peers to get a feeling of how was I being perceived by doing this kind of work, and they were alright with it, actually some wanted to learn and others already at my same or higher level. I want to stress that I am no math wizard, just a guy that does a state of the art review before proceeding with an assignment.

After this puzzling episode, I got a quite hard assignment. I did a bit of bibliography research and I found a paper describing the problem and proposing a solution. This solution involved some probabilistic math that I acknowledge that is not straight forward, but it got the job done.

After this I got called apart, this time by the company vice president. Same message; "Be average and lower the quality, you'll be happier, but we still expect great things of you"

I told the managers that it was very hard for me to do something "average" when I know the best. This is specially relevant because a part of my salary will come based on the financial performance of the tools I develop.

I also feel that if I deliver bad quality, I am not being honest with my self.

Any advice?

PS: I believe that all this comes because new methods represent more work for the company managers since they would have to catch up with them.

  • 7
    How much time are you spending on your "high quality" solution? It could be that they would rather have you use simpler, good-enough methods that take up less of your time. – David K Dec 12 '16 at 20:45
  • 8
    Their idea of "quality" and your idea of "quality" aren't the same. – WorkerDrone Dec 12 '16 at 21:00
  • 8
    And since this seems to be a translation, are you sure that they had a problem with the quality of your code rather than the complexity? There's a difference between "You're doing so well you're making us all look bad." and "The stuff you write is so 'elegant' that no one can make sense of it.". The latter isn't a compliment. – Lilienthal Dec 12 '16 at 22:00
  • 5
    If you really had a problem that could be solved with linear regression and you used multidimensional deep learning instead, the problem isn't quality. – Peter Dec 12 '16 at 22:49
  • 5
    Did they actually ask you to lower the quality of your work? Or did they ask you to change your work in a way that you perceive as lowering the quality? – gnasher729 Dec 13 '16 at 14:59
14

Some of the more theoretical advantages of simpler methods:

  • They are more often transparent (c.f. a neural net). It is possible to explain to a client why a decision was made for instance, or possible to reproduce the works yourself.
  • They are faster to train or run.
  • They are easier to verify.
  • They are more robust under slight changes in training data.

Some of the more practical advantages of simpler methods:

  • They take much less time to develop.
  • They are easier to review.
  • They are easier to debug.
  • They are easier to explain.
  • They work just as well most of the time.

If you don't understand all of these considerations in addition to whatever is state of the art in the field, then please, do not work for me or with me. To be honest I'm pretty sure something is missing from your story. There is clearly a way your work is not as high quality as you think and the answer is not "it's too brilliant." Shame on your management for not properly articulating this to you, but, kudos to you for posting here to try to find out what it is.

That being said, I do want to hire someone who:

  • has the knowledge to move our way of doing things forward
  • can teach hard things to coworkers
  • is jovial and fun to work with even when the work is hard
  • can balance many dimensions of concerns (i.e. the above list)

It sounds like you are adding this sort of value. But you are doing something wrong and you don't know what yet. Keep looking for it.

  • You may want to add a reference to Occam's Razor to this answer. (The professor in my first Machine Learning class referred to it as a guideline for model/method selection and I found that to be the most important take away message from that course. It's so easy to overfit on training data...) – AllTheKingsHorses Dec 13 '16 at 9:44
  • Fully agree, that simpler methods work better some times. In fact I have applied linear regressions for other problems. But the issue here is how to deal with bosses that feel intimidated by you. – user721807 Dec 22 '16 at 18:34
  • 1
    @user721807 I don't think they are, I think they're just annoyed the team has to maintain unnecessarily complicated solutions. – user42272 Dec 22 '16 at 19:29
  • I guess then excel with VBA's is easier than 10 lines of python – user721807 Dec 22 '16 at 19:30
  • 2
    @user721807 I'm just saying that for quite many involved questions like this it's best to be skeptical of the OP's narrative. – user42272 Dec 22 '16 at 19:37
3

Raise the quality of the team

Rather than lower your quality, raise your fellow peers' quality. Normally I would not suggest something like this, but you stated:

actually some wanted to learn

and

In fact I am idle some days while I get feedback

Having coworkers that are open and willing to learn from fellow coworkers is something to treasure. So for your peers that have indicated they would like to learn take that idle time you have and use it to teach/help/show them your tricks and techniques.

There is a second reason for this strategy. It is possible that your coworkers are trying to be diplomatic/polite with you more than being honest. Your peers may have said something to management or the VPs in casual or formal conversation about feeling intimidated by you, which is something they would never say to you. By trying to teach them it can help alleviate this or draw it to the surface that way you would be aware of it.

Third, if management continues to be annoyed with your higher quality, firing you would not solve the problem. After all you have taught many more people on the team to be just as much of a problem cough... I mean awesome.

Take note of other peers accomplishments

and others already at my same or higher level

If you have peers doing similar or better accomplishments with what they are working on take note of it. That way if someone pulls you aside again, you can cite other people's accomplishments as a defense to your own work. Saying something like:

I can see how me applying [Random important person's] algorithm to the data problem may seem impressive, but Bob just last week applied [Someone totally different's] algorithm to that other problem which from my technical perspective was far more challenging. In comparison what I did was rather average.

0

Here's a key line:

PS: I believe that all this comes because new methods represent more work for the company managers since they would have to catch up with them.

I imagine that the managers want you to develop methods which make less work for them. There is an ROI on their and your colleagues' coming up to speed on new methodologies.

Consider the benefits of your method ... does it benefit the company to use it? Are your more accurate results creating a better/safer/sexier/cheaper product, or just adding significant digits which can be ignored? Not being snarky here, it's an honest question which you should be asking.

So, let's say your method is better, saves the company money. How much money (time is money, yo) does it cost to train everyone and change processes? This is a comparison you should make.

Okay, you've done these calculations. Let's say that your way is better, saves money, isn't too expensive to implement. Now go and talk to manager types about it, armed with your analysis of the benefits.

I bet you'll have a lot more success pitching your idea if you can tie it to quantifiable company benefit rather than to your sense of professional pride. Seriously, why would they want to leave money on the table? Your job is to show them it's there.

-3

You are simply too smart for the company. When they said “best” it was only “average” to someone like you. Your machine learning knowledge was too good for the company.

Don’t waste your time. Quit your job. You will need to find someone like Andrew Ng to work with. Forget about linear regression, aim yourself higher - consider something like a self driving car system with a 10000 layers deep recurrent network. No training time, it just work with perfect performance.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.