2

Background
I'm a Data Scientist and am being asked to come up with a set of metrics/KPIs to assess my annual performance, and things like bonuses (and in the worst case being fired) depend on that. And of course I want them to align to my personal Data Scientist growth goals as well. My position entails:

  • working on machine learning projects from the start to end (getting data access, exploration, modelling, results, presentation, helping in productionising etc)
  • Analysing certain topics to give suggestions to the management
  • A part of which could be to educate stakeholders on using new tools to analyze on their own

What I've observed is that the job of a Data Scientist is rather complicated when it comes to measuring success, and things like performance appraisals metrics are then harder to design.

My Intuition:
On one hand, things like number of machine learning models deployed seem like a direct measurable metric, but on the other hand all models are not equal. Some may need a lot of time while others might be ready relatively quicker. The same goes for number of topics analysed/explored, as in exploration the required effort could vary a lot more than in modelling.

The only thing I can clearly put is the cumulative business value of the machine learning models developed, or of the business decision made from an exploration/analysis. This, however, puts the responsibility on the stakeholders a lot, putting an overhead work (to devise this value) needed to approach our team, and the stakeholders might hesitate approaching us. On the other hand, I believe that the stakeholders should be clear about the value of every topic they're working on, and the ones they requested other teams to work on.

Apart from business value, I also think about including something to assess the quality of every modelling project or analysis, something like a quality score, so as to drive me to improve the quality of my approach, to present better etc. But I'm not sure if quantifying the overall quality of such things could be practical and indicative consistently of what it was meant to be.

Question:
What could (generally) be good performance evaluation metrics for such a Data Scientist?
OR
What should the technical approach be to devise such metrics?

2
  • 4
    Step 1: define KPIs. Step 2: create model for how to optimise your time to maximise KPIs. Step 3: have conversation with management about how KPIs are stupid for this kind of thing. Nov 16, 2021 at 11:56
  • The problem is that metrics rely on a job being repetitive, so that the metric is comparable over time. Unless a job involves doing the same thing in the same circumstances over and again, then it's pointless to collect incomparable metrics. Although a job like data scientist may involve repeatedly tackling the same topic or repeatedly using a personal base of theory and experience, it is rarely the same work each time at a procedural level - the repeatable elements of most jobs are done by machine these days.
    – Steve
    Nov 16, 2021 at 13:24

3 Answers 3

5

Any metric can (and will) be gamed. You are asked to make up the rules for that game however, so you can just make metrics that you know you will be able to perform best on, and get your bonuses.

But if you want to create honest metrics, do not look at numbers too much. Number of models deployed? Sure, you'll just create 10 regression models and combine them into one ensemble model and boom!, lots of models. Number of projects? Not up to you, but just finish every project quickly without writing proper code and documentation, done.

So instead, look at actual competencies/skills and assess based on that. You want to measure improvement in those competencies. Schaun Wheeler suggests a few skills/competencies:

ELU suggestions

This of course needs to be tailored to your individual responsibilities, together with your manager. The possible skill levels here are:

  • Unconscious Incompentence
  • Conscious Incompentence
  • Conscious Competence
  • Unconscious Competence

Which is based on this model of Four Stages of Competence. These are not hard numeric values, but can be used to assess where your competence is right now and what you should work on. You can use whatever competence rating system you want of course.

Evaluate yourself now on these skills, and look at 6 months from now whether you've improved on any of these. Add new ones if you want to develop yourself further in a specific direction. Set a goal with your manager, e.g. "Bonus if skills X, Y and Z from Unconscious Incompetence to Conscious Competence".

You can still game this, but this way it is at least incentivized to become better at your job instead of better at deploying some number of models to make a cutoff. This does make it hard to actually evaluate your performance, since it is less absolute, but if you have a good relationship with your manager and you want to improve, it can certainly work.

1

It's always a good idea to stay simple when you write metrics when there are no clear indicators.

You need to understand your role as a Data Scientist, in the context of your job, and how it relates to the goals and outputs of the wider organization. The company's KPI sets out what is important for them, so you should be drawing from them.

You should be looking for a mix of lead indicators and lag indicators. A lead indicator gives on how your future performance will be and a lag indicator on how your past performance has been.

Good lead indicators could be:

  • evidence that your skills are staying current and that you're undertaking any necessary training
  • evidence that you are engaged across the business and are actively looking for and identifying new opportunities for data improvement

Good lag indicators could be:

  • surveys from colleagues on projects you have been involved in
  • individual performance against project timeframes and objectives
  • % of total data sources you have reviewed, cleaned, and optimized
  • % of total data sources where data accuracy has been checked
  • % of known missing data sources that have been identified and connected
  • if it's possible a metric on what value you've added to the organization through new processes, models, algorithms, or insights.
3
  • I feel that list is more oriented to being a Data Engineer, not a Data Scientst. Nov 17, 2021 at 0:20
  • @BarryDeCicco It's often repeated that Data Scientists spend like 80-90% of their time cleaning data. That might be an exaggeration, but there is a grain of truth to it in that data cleaning is a significant part of most Data Scientists' jobs.
    – nick012000
    Nov 23, 2021 at 1:34
  • @nick012000, I understand that. I've worked as a statistician for 25 years, and have dealt with a lot of data problems. Something like data cleaning/management is a core and expected competency; if they don't have that, they'd have failed out pretty early on. Nov 27, 2021 at 16:27
1

I'm assuming that a Data Scientist is not a Data Engineer (DE), and not an Analyst, but lives in the intersection of those fields. My criteria would be:

Business problems solved, short-term (i.e., one-offs) Business problems solved, long-term:

  • People in business units understood the solution.
  • Data pipelines set up as needed (possibly with DE supporting).
  • People in the business set up to understand, implement and manage the solution.

Business managers saying that you have saved them time/money/problems, and/or improved time/money/problems.

Training people to better solve problems (DE, Analysts, business unit people).

Demonstrating the you have gained the business knowledge to participate at a higher and better level.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .