Team relies too much on me for devops, which isn't what I want to do

Question

TL;DR: My team's planning and software development practices are awful. I want to do ML research but painted myself into a corner by taking the lead on development instead. Things are improving too slowly. Should I try to solve it or leave?

I'm a junior in a small R&D team. I have a computer science/data science degree, and my goal is to get machine learning experience to get into research positions, maybe get a PhD when I feel ready. I was employed with the promise of participating in published research. I got my name on a few papers but didn't get to participate often in the research side.

The team has very bad practices, which makes my job hard, as well as everyone else's. With my software development experience, I became something of an unofficial team lead by helping coworkers write better code, introducing unit testing, continuous integration and deployment, re-writing the deep learning code from scratch and taking ownership of it, among other things.

The goal was to set up better processes and teach my teammates, and progressively free up more of my time for my machine learning R&D job. Instead, the team started relying on me for deployment, software design, testing, etc. which I don't want to be responsible for. Everyone recognises the improvements but I'm still the only one pushing them. Planning is also chaotic, we constantly get pulled from tasks to work on urgent badly-planned deadlines.

The team's manager knows this, I've complained several times about it. They tried and failed to help; they struggle to get the team to change their habits. Then they tried to convince me to take the devops role full-time, I said no. One problem is that the official lead developer isn't interested in these improvements; they've been copy-pasting code to production machines forever and aren't responsible for the projects that require better practices.

How can I focus on the job I want to do? The situation is improving too slowly for my taste, but on the other hand, I might have trouble getting a pure ML position that fits my goals, since I'm not from a top school and don't yet have a PhD.

Answer 1

I think Joe Strazzere's comment is correct - if your current job isn't allowing you to go down the career path that you want, you need to either talk to your manager about getting on the career path that you want or finding a new job that will support you.

When you stepped up and started helping the organization improve their processes and build pipelines to support the development, you may have also pigeonholed yourself into that type of work within the organization. This type of work is necessary, but often needs people who are willing to do it. Finding those willing people could be difficult unless the company makes specific investments. The organization may have gotten extremely lucky with you - they hired you for one role, but you stepped up into a different role and were able to do it effectively. If the company were to switch you to the type of role that they want (and that they hired you for), they now realize that they would need to hire someone else to take on the work that you are currently doing.

This type of transition can be good when you can see how skills that you have and want to build upon can help the organization. This has happened to me - I've been hired into one role but found ways to show skills in other areas and make lateral moves and even promotions using skills other than the ones I was hired for. But when you don't want to use those skills, you could end up trapped.

The first step should be to have a conversation with your manager about a path to get you back to what you want to be doing. However, if that doesn't work, it is time to start looking for other opportunities.

Answer 2

You do need to make sure that your work is satisfying to you. If the devops work isn't interesting to you, although it sounds like you might be reasonably good at it, you do need to make it clear to your current manager that your interests lie elsewhere. Try to avoid making it an ultimatum, but you should try to make it clear that you are considering leaving the company altogether. Your manager may be aware of your skills and adaptability and understand that you can be effective in a different role more aligned with your interests.

Another consideration is, suppose this employer offers to move you into a different role, would you really consider staying? The workplace sounds a bit chaotic, maybe the manager isn't very effective, and maybe all of that is due to a negative company culture - do you really want to continue to be part of that? Or is there something valuable for you there that makes it worth staying? I think these are useful things to consider and once you've made your decision, no regrets, you made the best decision you could with the information you had at the time regardless of what happens in the future.

Answer 3

Your manager already seems to understand the value of the more rigorous approach to development and deployment that you've introduced. I would go to him next to explain that this approach, being currently reliant on essentially a single person driving it, is also currently very fragile because it currently has a "bus factor" of one: if anything happens to you, all of this probably starts backsliding and falling apart.

Suggest to him that to mitigate this problem you need to start training others in your organisation to develop the skills and habits of continuous improvement in these areas, and that this would best be done by getting a developer or a small group of developers (ideally volunteers) to start taking the lead on these things, with you as an advisor/coach/trainer for them.

The idea here is that this person or group becomes the one leading the identification of problems, bringing up the "how do we deploy this?" questions, and the like, and having an initial go at coming up with answers for these things, though you will be providing them a considerable amount of guidance and design and implementation assistance at the start. You should, however, provide them not immediately with the answers they need, but guidance on where to look for answers, and help following that guidance. This should change over time from the very specific ("think about changing the design of X module so that a test like Y could be built," followed by fixing whatever design they come up with and even pair-programming it with them) to the much more general and hands-off ("can you use a technique similar to Z that we used over there?" followed by letting them do whatever they do and only then reviewing the implementation).

This approach has two advantages: it gets this stuff into the development group's culture so that it happens "automatically" rather than eating up your time driving it, and it presents the move of responsibility and work off you as something that's being being done for the benefit of the team and the manager, rather than just for your benefit.