too much pressure at work in the machine learning area

Question

In short

I'm a single data scienstist.

I've joined recently a team of 4 python developers / machine learning engineers. The machine learning project we do is difficult as a machine learning problem. We are also expected to test a particular new technology on it. No one in the team has used it before. Moreover the team uses a lot of tools I had no idea about (e.g. for dealing with dependencies or managing resources in a cloud). I feel that all of this is too diffucult to me. Nothing goes smooth.

In the last few sprints I worked overtime to finish everything I was expected to do. I constantly feel incompetent and I'm getting depressed by the Monday's lunchtime. This affects my job performance, since when something goes wrong I'm getting stressed and I can't think how to solve the problem.

I don't use my knowledge at work and I don't do stuff that help me grow professionally.

Right now we work on the data preparation but soon we will have to build a model. We will probably do this is a new version of a machine learning library I haven't used before. I have no idea how to do properly a project using it. I feel that the team expect me to know it and to lead in this subject.

Working from home due to COVID-19 doesn't help.

I was always a good performer (during studies, in all previous companies and in the same company in different team). I liked my previous position in this company very much just a year ago - I was very successful, got promoted and at the end of holidays I felt I miss my work and I'm happy to go back to the office. However right now I'm constantly feeling depressed and I'm struggling to find any positives about the whole situation.

What should I do?

My only idea is to finish our sub-goal, then take one month of unpaid leave to learn the new version of the library, distance myself from work and go back. There is no strict deadline on this project. This would help me grow professionally and make me more competent and confident in the current situation.

More details

• There is one guy in a team who seems to be especialy unhappy with the code and things I do. When he reviews my work there is a lot of issues he points out, and after that he provides ways to improve our processes, which are usually ideas of more things I should do. Reviews of my code take around 2 days, sometimes he suggests some solutions I had no idea about, I'm trying to figure out how to do it, then later it turns out it's not possible or he critises something, I explain myself and after that he agrees that it's agreeble work-around. I feel like this guy assumes everything is just wrong and doesn't spend much time in thinking why I did it this way.

• There is a big difference between my and team's knowledge Just today I spend 1.5 hour learning about a tool they want to use (possibly in the machine learning project I do) to prepare for a 45 minutes meeting about it. I said 2 or 3 not that stupid (I hope so) sentences in the whole meeting. However I didn't understand all issues they discussed. At the end of the meeting this one guy asked if me and other guy (ho didn't say much) "have anything to say / summarize". This got me really angry, since I spend some time trying to understand eveything and barely was able to follow them and I felt that it was not enough and he expected me say more at the meeting. I was already trying to avoid similar meetings - but there were some concerns that in a team we all need to be on the same page and avoiding meetings won't be good.

• Our aim is also to test new technology on it (Airflow) - I hadn't used before, but this is something I'd like to learn. There are also some tools that I have no idea about. This makes me very unsecure and unhappy at work. Normally I'd just learn similar stuff in my free time. This situation is different because it's not what interest me (this libraries are not about machine learning itself but more about python / cloud development) and it's not aligned with my long term personal goals. I already work on a private project (more around research and theoretical machine learning) after working hours and I just don't have more time to spend on professional development.

• Due to some corporate restrictions (which departments do what) not all projects (including easy and nice ones) were allowed. When I joined there was a list of ideas for a project that could be done. So we picked the one that have looked the most promising. However it's very difficult problem and I'm not sure use if the model will work at all. I communicated it. I still feel that if we fail it will be my fault, either because I should have pushed for getting the easier projects or do better model to make the project work.

• If I leave I'll lose an investment which is now worth around 1.5 year of my work. Ideally I'll have to work 9 months more here to be able to get it back.

• I feel that I'm getting stupidier at work When I graduated and started working in machine learning I felt I was good. Now I feel I'm spending a lot my time on pointless meetings or dealing with some internal problems, not on machine learning. There are new algorithms / models and libraries constantly developed. I feel I'm not able to keep up with this pace or even be able to use the math / models I knew because for 90% of my time at work I do other stuff.

Edit

• Meaning of "If I leave I'll lose an investment which is now worth around 1.5 year of my work" - there are some financial benefits (shares) provided by the company that I'll be able exercise / get money from in 9 months. If I leave earlier I won't get them at all.

• How far are you into your career? I'm a senior - but my position is different that rest of the team and I'm assumed to know less in the area of their expertise.

• Why is what you're doing not in line with your career ambitions? I'm in this company purely because of money. I though I'll manage to stick there for less that a year now but not it's getting very difficult for me.

• Why would you take a month unpaid leave to study something you're not interested in? The tool I want to study during the leave is interesting to me. Uninteresting tools are the ones we use right now and Im having diffuculties with. I want to do it because this is something specific to my position and I feel I was hired to teach the rest of the team best practices concerned with usage of this tool. However the field is changing quickly and I'm not up to date with it (I used this tool one year ago in my work).

• Have you been told your performance is an issue? No, we get 'grades' each month and I got the best ones from 2 team members, and average from the rest.

• Wouldn't learning about this be just a part of the job just like it is for the rest of the team that are new to this? Yes, but I'm the only one that doesn't know most of the tools.

• Why don't you want to just grow into your role and deliver instead of pursuing what seems like an elaborate scheme with no clear goal? Because in order to deliver right now it seems that I have to become more like a software developer and loose my skills as a data scienstist / machine learning expert, i.e. get similar to other team members. I don't want to do this. Since Id like to grow in data science / machine learning area.

Answer 1

I think your problem is rooted in your own insecurity and probably unrealistic self-imposed expectations.

You recognize this yourself. However I think your approach is totally the wrong one for this situation. You are a 'data scientist', your skills are different to the ones of your colleagues and there will a moment where they will rely on those skills to do the job needed to reach your company's goal.

We will probably do this is a new version of a machine learning library I haven't used before. I have no idea how to do properly a project using it. I feel that the team expect me to know it and to lead in this subject.

This worries me a bit. You may not know how to do something with that library, but you should certainly have an idea on what to do with that library. This is what your colleagues don't know and they will be relying on you for that.

I feel personally very close to your situation. I'm a CS Engineer and all my colleagues are aeronautical engineers, physicists, etc. I have very limited understanding on the topics they work and sometimes it is frustrating to see how easy it is for them to see things I can't even fathom. However we need to put their work into maintainable tools, traceable requirements and other things I am good at. And they rely on me for that.

What you should do is to embrace your ignorance. It is fine if you are not as good, just try to learn as much as possible and help in what you can. Don't be defensive. Projects have different stages, it looks like in the current one you are being kept 'on hold' and your moment to shine will be later on. Brace yourself, try to help them and just wait a bit. The learning will be useful.

Not everybody get to work all the time in their narrow field of expertise. Sometimes you need to work in multidisciplinary fields and the situation is like the one you describe. For me, that's a very rewarding job: you can be learning now from them, they will be able to learn later on from you.

Answer 2

The way I understand you problem is that you need to be trained in some technical areas. The most reasonable thing to do is to talk to your manager about this and ask for support. Depending on the specifics of the business, your manager should be able to find a good solution for you, possibly with the help of your colleagues.

To be even more professional, before you talk to your boss, spend some time alone to understand your problem exactly - including the exact areas for which you need to be trained. This information will be very helpful during the discussion with the boss.