How can I best ensure applicants have requisite foundational knowledge?

Question

I am in the process of hiring my first full time data scientist in an academic setting. The position is entry level, and I expect to personally provide mentorship for the skills specific to my field that they will need to succeed. My organization is large, and they will also have support from other groups.

However, I worry that if the applicant I decide on does not have the requisite foundational knowledge that I will spend too much time helping them learn these skills. For example, I would expect the applicant to know basic joins in SQL, how to write a basic program in language X, and how to use Git. These expectations are clearly delineated in the job description, and the candidates I have all claim experience with these skills.

Tentatively, I am planning to implement a short (approximately 30 minute) assessment exercise that will completed during an in-person interview. I will encourage applicants to use online resources (such as Stack Overflow) and also encourage them to ask me any questions. I will warn them in advance that they will be asked to complete the exercise. However, I have read a number of experiences on this site as well as this Q&A that indicates these types of tasks are sometimes not well received. I worry that this type of assessment could be stressful and off-putting for applicants.

I have nearly complete control over the interview process.

How can I ensure applicants for an entry-level position have the requisite foundational technical skills while not scaring them off?

I am also open to a frame challenge. Perhaps I am approaching the problem incorrectly and the more important quality is attitude and ability to absorb new skills.

Answer 1

This is a fairly standard concern, and there are a number of ways of tackling it.

Obviously the first thing you do is filter resumes, removing anyone who doesn't list the skills you want. I usually prefer people who have actually used those skills in a project, but that may be harder if the skill is incidental to their main job.

Then there are number of ways of assessing technical competence. In increasing order of effort:

1. Technical Questions. Ask in the interview questions about the language/system that you expect people with the required competence to know. In your case SQL I would simply give them a problem - i.e. define some tables, and ask them to write SQL to retrieve certain records such that they need to use a JOIN. This should be doable on a whiteboard in five minutes or so. Don't worry about syntax errors that would be picked up by a good IDE. I use this approach to check language knowledge, and I've never had anyone who answered satisfactorily and didn't end up being competent.
2. Short Coding Exercise If you really need to check language skills, then have the candidate write a short piece of code in the appropriate language. You can do this interactively on a whiteboard, giving them the problem, or you can leave them to do the exercise on a screen. Google uses the first approach. Using a screen can mean problems if the candidates isn't familiar with the development environment. I failed one once because it took me a significant amount of time to get the environment working. This can be longer than the example above, and longer than fizzbuzz, but keep it down to 30 minutes or so.
3. Online coding exercise. Platforms exist where you can set candidates coding tasks for them to do in their own time, but time limited and taking an hour or two. These are really for positions where coding is the main activity, not data scientists, and even then some people dislike taking them. If your company doesn't have a system like this it will not be worth setting one up.
4. Take home coding exercise. Here you get given a problem to code on your own time over a few days. These get a really bad reputation and many candidates will decline them - especially the good ones. If you are a candidate with multiple promising applications progressing you won't waste time with the ones who want a lot of work from you. It's a different matter if your company is extremely desirable of course. This is definitely not the right approach for a position where coding is not the main activity.

For your position I expect that the first option, and possibly the second, are appropriate.

Do remember that a candidate who has used SQL, but not for a number of years, may struggle to write SQL in an interview if they are not warned in advance that they will need to. However a person like that will probably come back up to speed very quickly, so don't necessarily fail them.

Also tailor your interview to the job duties. If the employee will be doing 90% data science and 10% coding, don't make more than 10% of the interview etc. about coding.

Answer 2

I'm going to flip this on its head.

You're bringing a data scientist on board. Their role should be more geared towards the data science portions of their role and less towards the fiddly technical bits of their role.

Not all data scientists know all tools. One could reasonably expect a data scientist to know a thing or two about R, Pandas or NumPy (not necessarily Python since you can always write Juypter notebooks which are Python-like anyway), but the movement of data and storing it into a database and the like is a bit of a reach.

In doing interview assessments, it's important to keep this in mind.

Why am I bringing this person on board? What do I value from their participation?

You could scare off a data scientist with 20 years of experience in the field because you're asking them to do something that they have never needed to do, versus getting someone with less than 2 years of experience in who has bigger aspirations of being a generic programmer because they know all these things.

---

Ultimately though I agree with DJClayworth's assessment. You do need to put together some kind of assessment packet that satisfies the questions you want answered. But I stress this: ask questions that value the actual position, not just your preferred nice-to-haves.

You can always teach someone how to use Git or how to craft SQL if you had to, and you should realistically budget some time in your onboarding process to do that, if nothing else but to at least get them familiar with your schema and Git repo layout.

If you spend too much time, well you need to quantify what that is. If an applicant isn't able to do what you're asking of them after about 3 months, then that's reason to terminate and re-start the search. But you should go into this process with the expectation that you'll be asked questions about these kinds of things for a period of time.

Answer 3

Perhaps I am approaching the problem incorrectly and the more important quality is attitude and ability to absorb new skills.

This would be my inclination. I would perhaps broaden it to some other factors, but I'd keep it at this higher level, rather than focussing too much on specific technical competencies like "can write a join in SQL".

Unless you have a very specific need for somebody who's ready to hit the ground running on an urgent SQL-joining project, looking at a specific narrow competency like this may not be a great indicator of overall capability. As noted in Makoto's answer, there are a great many data science languages/tools out there; just because somebody hasn't worked with SQL yet doesn't mean they're incapable of picking it up, and just because they've spent the last ten years working on a SQL project doesn't mean they're going to be great at anything else.

An approach that I've found effective for data science interviewing is to present candidates with an open-ended, language-agnostic scenario and get them to talk about how they would approach it, leaving it broad enough that any DS candidate ought to have some relevant ideas, while making it just specific enough that it's unlikely to be completely covered by a generic answer. For instance, here's a question that we used last time I was doing DS recruitment. (NB this was not an entry-level position, so we were a bit more demanding than you would want to be, but the general approach should translate.)

Computing: The purpose of this question is to test your knowledge of computing techniques that may be relevant for data science work. We're not assuming any specific programming language but you're welcome to discuss how different languages might affect this scenario.

Several years ago, [our org] implemented a new mathematical method for producing key economic estimates that runs on internally developed software. Recently, the amount of input data available for these estimates has greatly increased, and the run-time has become unacceptably long. You have been asked to find ways to speed up the software. You can rewrite the software as needed or transfer it to a different environment, but you can't change the underlying method - the results produced must be 100% consistent with the old version.

What options would you explore to improve the run-time of the software?

There are many different ways this could be answered, depending on the candidate's background, and I wouldn't expect any one candidate to cover all the options. But "how make computer go faster?" is a perennial data science problem and any good candidate ought to have something to say here.

Once they've made some suggestions, we can then dig further in those areas to explore their competency further. For instance, if a candidate mentions changing language, I might ask them to explain why they expect Language X to be more efficient than Language Y, or about what the risks are in changing languages and how they could be mitigated. Effectively, we're starting by identifying where their existing experience lies, and then exploring how they can show competence within that experience, rather than testing on one or two specific points of knowledge.

The "must be 100% consistent with the old version" requirement is here because we don't just want candidates who know a bunch of different methods; we want candidates who can think critically about which of those methods is appropriate for the problem at hand. For instance, we had some candidates who suggested changing the estimation method; this might have solved the speed problem, but it didn't fit the problem requirements.

In addition to this question, we also had one that was a bit more on the data science side - here is a situation with an information need, these are the sources that are available, talk us through how you would approach this. I don't have a copy of the question text handy, but it was the same kind of strategy: pick something broad enough that any good candidate will have something to say about it, then based on their initial answers dig further.

Answer 4

Specifically when you check if someone can use git: It takes quite a while to setup git, or to get used to the way it has been set up. Likely more than 30 minutes. Once that is done, tasks are a lot easier. So a 30 minute test doesn't work very well.

What would work a lot better is to ask them what you can do with git, about the principles, everything to do the basic work. Being forced to talk about it makes it hard to make up non-existing knowledge on the spot.

Answer 5

Entry level could mean BSc or MSc in Data Science or something else, perhaps a graduate Journalist with major in history, becoming a scrum master then an expert in programming over time.

Match your expectations to the goal, long term or short term? SQL major or minor?

Computer Science skills are primary to the job but so may be Business Analytics and university marks ought to have been attached or requested ought to be a graduate of Data Science. If your company goals are soft on marketing analytics vs software development, overlook SQL but look for majors in other software languages that show aptitude. Mastering one or diversity in many both counts as "points".

Is SQL just a routine "scraper" for other apps to manipulate? Can those tasks be automated?

To this end, grade each of your requirements with a weighting value * experience then add up the scores. Choose at least 6 (see below.)

Like bidding cards in Bridge, are you willing to pay for a strong hand or just looking for sufficient majors worth more points?

Analytical skills. Computer and information research scientists must be organized in their thinking to evaluate the results of their research.

Communication skills. Computer and information research scientists must be able to clearly explain their research, including to a non-technical audience. They write papers for publication and present their research at conferences.

Detail-oriented. Small programming errors could affect an entire project. How do they design self-checks/tests? afterthought or during design.

Interpersonal skills.

Logical thinking. Ask what are their favorite algorithms and why.

Math skills critical to computing. Ask about neural networks, non-linear algebra, Fourier Spectrum

Problem-solving skills. Ask about their most innovative solution

The mathematical product sum of each requirement ought to select your best candidate, unless you want to change importance factors or add personality types like Meyers-Briggs or self-motivated introverts or task-following extroverts.*

Answer 6

DJClayworth has great ideas. The other idea that comes to mind is to have them do a technical interpretation of a sample of your project.

For example, provide them a snippet of code (basic table structure and query) and ask them what it does. This will ensure that they either know what it does and have the requisite knowledge or can figure it out and will be able to gain the required knowledge (many frameworks, databases, etc have similar syntax that provides them enough info to get started).