I really thrive off learning new things and figuring out how the component pieces fit together. When do you use X over Y? How do we make Z more efficient? I don't want to be an SME. I want to be a jack-of-all-trades who can do a pretty damn good job in a lot of areas--can get a lot done myself, but I also know when to lean on the SME. An architect, maybe.
I've been working the past couple years in a data engineering/data munging capacity (pandas + Spark et al.). I feel like I've wrung all I can from that. More recently, my team has transformed such that I'm doing more backend web development (APIs, Flask, AWS infrastructure, etc.). I enjoy those elements, a lot, because figuring out what technology is appropriate and connecting the pieces together fits into that broader architectural piece I enjoy. I think ultimately I would like to do application development.
I was approached by another team about a machine learning engineer opportunity. They want to build a library to simplify and productionalize models so that our data scientists can simply focus on the statistics aspect. e.g., instead of 20 steps to handle hyperparameter tuning or RIM weighting, they can just call one function. No boilerplate. It's very little about the math and mostly about software architecture.
I have no interest in data science, honestly. But I'm not happy in my current role (PO is a nightmare), and I think there is intrinsic value in learning how to design libraries and architect code in a meaningful way.
With that said, given my goal of being a jack-of-all-trades application developer, is it likely pursing will pigeonhole me into data science-adjacent pursuits, especially given my data engineering background? Or is it likely the experience will be transferrable?