Interview Question about familiarity with a programming language

Question

I've done a rash of interviews lately. I have a pool of questions I generally operate on. One of them was "rate your experience with X language from 1 to 10." Well guess what? EVERYONE rated themselves a 7 or 8. Both people with no business doing so and the 20+ years experience guys would say 8. So I dropped the question and attempted to rephrase but it just ended up confusing.

I generally still ask questions that give me an idea on their experience but is there a way to ask this question in a meaningful way or is it just a bum question to ask?

Answer 1

is there a way to ask this question in a meaningful way?

Yes.

Your observation that everyone says 7 or 8 is spot on. Use that. What I would do when I asked this question in interviews is follow it up with "what is something you had difficulty with when you were a 6, but find straightforward now?"

This then leads into a discussion with the candidate about where they are at, what they still find challenging, and so on.

Remember, every interview question should be designed to elicit a clear signal. You are right to critique your question for producing poor signal. Think hard about what signal you wish to elicit from the question and devise follow-ups that lead naturally to a conversation that gets you that signal.

When I was interviewing for positions myself, I had a prepared answer in case I was asked that question. If someone said to me "how good a C++ programmer are you on a scale from 1 to 10?" I'd say something like

I'd rate myself a six; I have extensive experience writing compilers and other language tools in C++, with a particular focus on COM programming. I have a solid grasp of the language fundamentals including preprocessing, lexical analysis, grammar, semantics, and what behaviours are undefined. However my knowledge of the C++ standard library is comparatively weak, and I have never written a C++ compiler. Nor have I been on the standardization committee for C++. When I worked at Microsoft I had people down the hall from me who had worked on C++ compilers and made significant contributions to the language design; those people are tens, and I am not them; it would take me many years of work specializing in C++ to become a ten like one of them.

That answer was designed to both give an accurate summary of my C++ skills, and to give the interviewer many opportunities to say "tell me more about..." on subjects that I was prepared to discuss extensively.

Answer 2

You asked,

is there a way to ask this question in a meaningful way or is it just a bum question to ask

I don't think there's a meaningful version of this question.

Questions with simple, numeric answers (I'm a seven!) seem attractive because they're easy to ask, but they're not always meaningful. Ultimately, the core flaw in this question is that it's a self-evaluation with no relationship to the actual needs you're hiring for. My perception of my skills may not match up to the general public's perception of my skills, even if we can agree on a generic scoring criteria. And as you're discovering, many people will rate themselves "too high" - or, at least, the distribution of self-selected scores will be skewed too high. Just go sit on a busy street corner and ask everyone who drives by, "are you better at driving than the average driver?" and I will bet that more than 50% of people think they're better than average.

Perhaps more importantly, asking someone to "rate their experience" has essentially zero relevance to what your specific needs likely are. For instance, I've hired for a team of SQL developers who were tasked with building a large and complicated data warehouse. Yes, they used SQL as their primary language, but we had pretty specific needs around specific types of problems. Instead of asking candidates for a generic SQL rating, I would focus on those problems. And instead of asking for ratings at all, I would focus on getting the person to tell a story. So, instead of asking,

Rate yourself on SQL

I would ask,

Can you tell me about a time you were writing SQL code to work with a large block of data? How did you make decisions about where to look for performance improvements?

Or,

Let's say you need to build a data model that will have transaction data loaded via a batch job every night. Then, throughout the day, users will be running reports on the data. How do you decide on an appropriate level of sturcturing and processing to do during the batch job, versus doing things at runtime when reports are requested?

This way, you get the candidate to actually talk about their skills, and talk about problems they've solved - or how they would approach theoretical problems. While some people will try to fib their way through these questions, it becomes fairly easy to spot people who wouldn't be able to accurately self-assess. It also gives you a lot of context details about the person's thought processes. Anyone with google can find all kinds of reference material about how to write SQL code; I don't expect developers to memorize everything in the language. Basically, I'm not trying to find a literal SQL expert. Instead, what I'm looking for are people who can use SQL to solve the specific problems I know I will have on my team. Getting a candidate to tell stories or explain their thought process is much more meaningful for that kind of evaluation than asking them to self-assign an arbitrary numeric score.

Answer 3

Let me give you a personal answer on this.

"Kevin, how good are you at C#/.NET, from 1 to 10?"

"8." I mean, yeah, I'm good with the language. But I know that I haven't done a lot of stuff with Moq, and I'm a bit weak with Unit Tests and TDD. Plus, well, I haven't actually used Resharper - isn't that a critical part of good standard code in .NET? I don't even know enough to answer that. Yeah, I've done a bit with Generics, but there are loads of people that have, and they're able to use it much more intuitively. Plus, a lot of the stuff I do that's more esoteric, it's not because I completely grasp it - it's just a general knowledge of what a StackOverflow.com answer told me. "Yeah, definitely 8."

Sit me down next to another '8', though, and I'll be saying, "Wait, why are you making that a public method? You shouldn't be accessing it outside the class at all. And why not remove this side-effect of it unnecessarily changing a property of the class - remove the side effect, make it a pure function, and swap it into a private static? Plus, you shouldn't be creating two versions of this method - all they differ on is which derived class they operate on. Make a combined one that's either in the base class or write an extension method that can operate on the base class. DRY - don't repeat yourself."

Do you see the problem?

When you asked me what my rating was? I thought of all the stuff I'm aware of that I'm not an expert on, that I'm weak on. Moq, Resharper, TDD, etc. So I know I can't give myself 9 or 10, because wouldn't a '9' know that sort of stuff down pat? But ask the other person what their rank is? They're not aware of what they don't know. They've got IF branches, FOR loops, and can intuitively use List<>, baby! 8 for sure!

Basically, you're getting screwed over by both the Dunning-Kruger effect and Imposter Syndrome. The experts know how much they don't know; the less experienced don't, so they think they know more than they do.

Edit:

So, this answers part of your question - is it a bum question? Yep, it is. So what do you do instead?

You separate the '8's that know what they're talking about from the '8's that don't. You come up with some questions that you'll get different answers between those two groups. When I was coming up with interview questions for the last two hires, I chose to create two programming problems - one which was relatively simple, one which was a bit tougher and a bit more open ended (there were several routes to go, and part of the problem was whether the applicant knew enough to clarify the situation before writing the code) and a third code-review problem which was "here's some bad code - what all is wrong with it?"

But ultimately, this is the entire point for all those technical questions/whiteboardings/etc: does the applicant actually know their stuff?

Answer 4

I'm guessing you are looking for a way in which you can easily filter candidates, not actually know their level (you can do that later in the interview process). In that case, the fact that beginners and experts rate themselves the same is a problem.

I'd suggest a deeper dive into the language--it's here that you will find who actually knows what they're talking about.

Ex:

1. Never heard of
2. Seen but never written
3. Written, but unfamiliar
4. Written and comfortable
5. Written and can explain deeply.

Please rate your C# experience with the following:
Generics: ①　②　③　④　⑤
LINQ: ①　②　③　④　⑤
Inheritance: ①　②　③　④　⑤
Events and Delegates: ①　②　③　④　⑤
Etc.

You'll find that the scores are still generally high, but you should also begin to see a separation based on years of experience. In this way, you'll actually be able to filter your candidates beforehand.

But of course, to write a decent survey like this is tedious and would require someone with solid knowledge to begin with.

If you just want a pass-fail system, you can use an online coding test. I think Linkedin even has their own "certification" system

Answer 5

When I interview tech staff I never ask questions like that, because as you have found they are meaningless.

How do you rephrase this question?

"so, tell me about your experience with x language"

Then let the candidate speak without interrupting them.

From their response you should be able to tell what their experience is, and you can ask follow up questions to things they mention.

The key is to let them do most of the talking. If they don't have much to say then they don't have much experience.

Answer 6

Sorry to add yet another answer, but this is important and no one else brought it up yet.

You are failing to consider the question from the perspective of the interviewee.

Interviews, particularly for candidates who went through lots of formal schooling, are a giant game of "what answer do they want"?

Thus the candidate goes through the mental process of "well, if I say I'm weak they won't hire me, but if I say I'm awesome they'll ask me to prove it and I can't, so I'm a 7". If they have some actual confidence in their skills they might say "8".

It's not that those numbers are accurate, it's that they're defensible: the candidate likely knows they're not really a seven but when pressed feels like they know enough they could defend that assertion. But now the conversation is a philosophical one about the meaning attached to a point on the scale.

Another driving force here, and I mention schooling for a reason: in school 7-8/10 is average. We can complain all day about grade inflation but we all know it's a thing. We've trained the last 2-3 generations to think that way. Someone today under the age of 45 (full disclosure, I'm in that < 45 group) would likely simultaneously say that they're a 7 and that they're an average programmer. To that person those mean the same thing. No way is that average programmer going to rate themselves a 5/10.

Eric Lippert can get away with saying "I'm a 6" by virtue of being Eric Lippert: he's already a known quantity and has his history of accomplishments and CV to draw on (apologies for riffing on your answer Eric). He'll never be sitting in an interview being subjected to a cold read. But for the mortals they won't risk it and they'll play it safe, assuming that they don't just consider 7 average in the first place.

So I recommend ditching the question: it might be salvageable but your first thought for any interview question should be "how are they going to (mis)interpret this" and that question fails that basic psychological sanity-check.

Answer 7

Any self-rating will suffer from the fact that it's only possible to compare something you know to something you know. Since they probably don't know any language to a firm 10, they can't compare to an unknown. The most famous example of this is the Dunning-Krueger effect.

Instead, ask them to rank their knowledge of multiple languages from best to worst. This will give you an overview of their knowledge.

To get an absolute level, ask if they can do a specific task. Elaborate from there: get a time estimate, ask how exactly they approached the task last time, and get into what they would do differently with your task.

E.g., "We need a REST API that receives a JSON that looks like this and stores its contents in Oracle XE. How many days does something like this usually take you?

Then ask about the detail: which frameworks and libraries they will use, how they will approach this and that.

A good developer (middle+ level) will have questions of their own, e.g. which logging they should report to, is the service to be containerized, and so on. If they don't, they're probably not a 9, but you can raise the questions on your own to see if they're familiar with these aspects.

It's best if it's a task that has been done before in your company, so you have a basis for comparison.

Answer 8

Rather asking them to self-evaluate, which I doubt anyone could do completely objectively -- especially in a situation where they want to sell themselves -- why not get candidates to demonstrate their knowledge?

Let's say you're asking about proficiency in Python; keeping to a fairly core mechanic, to cover most bases. For example, say you focus on OOP: start with a mid-level question that probes their knowledge of, say, MRO. You could then level up (metaclasses, etc.) or down (inheritance vs. composition, etc.), depending on their answers. That way, you establish for yourself their proficiency -- in an area that is important to you (OOP in Python, in this case), as you control the narrative -- through evidence rather than subjectivity.

Answer 9

The problem with this question is that it has no proper definition of what a "10" is. So everyone will interpret it as they want and give you the answer they think is the highest rating they can get away with without sounding arrogant.

So you might get much more meaningful results if you quantify what different ratings would mean:

How would you rate your experience with X language from 1 to 10, with 1 being "I wrote a Hello World program once", 5 being "I know all its features" and 10 being "I wrote The Book about it"?

Answer 10

I agree with most of the other comments on the inadvisability of just taking numeric ratings, but I think there is another aspect that is only touched on in @Mars answer. You are asking them for an overall rating in a vast area "language X". In the parts of the language the interviewee uses everyday they may be a 9 (or better but there is always something you need to learn). On how to use it for producing (or talking) to User Interfaces they may only be a 6, for working with databases they may be a 3. There may be other areas that they don't even think about. So how is an interviewee going to group all of those into a single number? They are going to average them (with some bias towards the high numbers as if they think your job was worth applying for, it is hopefully going to be in their core area). Even before you change from "rate your experience" to "tell me about your experience with", stop and think about what areas of proficiency you really want to know about and address those more specifically.

Answer 11

I recently had an interviewer ask this type question, so I explained how I looked at the rating system as a exponential system, rather than a linear scale.

The top of the scale of "10" are the devs who wrote the language, library, or technology. A "9" being someone with roughly absolute mastery of that technology, like someone with 20+ years of experience and has dealt with "everything".

An "8" would be someone who is at least a Senior position with lots of experience, probably a fair amount of formal training, and works on it on their own time. A "7" would be a mid to high level programmer that's dealt with a fairly wide range of problems over a handful of years.

A "6" would be getting into a Junior level dev, maybe right out of college that understand the quite a bit, but hasn't really done anything professionally.

Below that get into more basic understandings of parts of the languages, etc. Does the person understand the concepts of loops, collections, objects, APIs, etc.? A complete beginner (without any relevant or related knowledge) would actually be a 0 and should be able to progress to about a "3" in a year, with another 2-3 years getting to be a "5" or "6". Someone coming from a similar technology could get to be a "6" in a month and a "8" in a year or less, such as if they switched between Java and C# plus had an "8" level in the complimentary language.

So you can see that my view is completely different from what others see in the scale. I'd view myself as a "7" in my current role, but I've talked to some people who would think I'm a "9" and other's would think I'm a "5", due to the relative scale when comparing me to themselves.

Because this is so subjective, the scale definition varies so much, and you have the Dunning-Kruger effect (as mentioned in another Answer), you'll never get a good answer to this type of question.

You really do need to ask questions about what kind of things they've done before and how they solved problems. Then ask why they didn't choose a different method.

You: So, you solved this problem using an array, but why not an List?
Them: What's a List? {Definitely not the 7-8 they said they were.}
Them: Well, I didn't need all the extra methods and features that would have taken up extra memory and processing power. I just needed something simple to pull out pieces of data easily. {Probably the 7-8 they said they were.}

This is still subjective, but now it's you being subjective rather than the candidate. As long as you can control how things are evaluated, you'll have a better way of knowing what it is you really want to know. Sure, someone might be able build a Quick Sort from memory, but can they build a complex data structure and have the Quick Sort iterate over an ICollection of them?

Maybe you don't need an "8" and are willing to train a "6". This comes down to understanding your own needs and wants, and finding the right candidate for it as well as gearing your questions to those needs. A standardized test won't work for that. A "9" might breeze through it and consider it a waste of their time, while a "6" struggles and gives up. Same way with a standardized interview. If a "9" isn't challenged by the interview questions, they might not think it's worth getting hired, since they won't be challenged by the work. A "6" struggling to come up with answers during the interview might also give up there, due to thinking the job will be the same way. Not many people want to have a job where they are only utilized for 50% of their effort and knowledge. Nor do people want to have to spend 120% of their time, energy, and brainpower learning new stuff for the job.

So what I'm saying is that the interview needs to be geared towards the candidate, not just that you have a list of questions you ask everyone. You can have a list, but don't robotically ask every single question to every single person. It kind of sounds like you've got that already, so I'll stop here, since I think this is getting off track of your original question, anyway.

Answer 12

What mostly matters is how long they used a language on production environment. Therefore you can ask this:

Have you ever developed any project which ran on production in X programming language? If yes, for how long?

Then guess the number by yourself based on how long he/she used the language on production.