Are puzzles an effective part of the recruitment process?

Question

Many companies - particularly in the tech sector - use puzzles of various kinds (including riddles, lateral-thinking questions, and guesstimation questions) in job interviews for potential new recruits.

To what extent is this practice effective or ineffective? Is there any correlation between puzzle-solving performance and job effectiveness?

If there is evidence of correlation, then for what types of roles? And what types of puzzles?

And if (as I currently understand to be the case) there is little or no evidence to support the effectiveness of puzzle-based job interviews, then why do companies persist with this practice?

Update: Thank you to everyone who has posted answers so far. Mostly what we have at the moment though is opinion, personal experience and anecdote - are there any actual data on this? I.e. studies?

Answer 1

First, a disclaimer: when I interview people I do not set them puzzles. I used to and I stopped, because they gave me false positives and false negatives. I never used a puzzle from a book or the internet though. Second, some perspective: chefs, nannies, accountants, heads of HR, and university presidents are not asked these things. It's mostly software developers.

Now, why set a puzzle? Many people think it's a proxy for many software development tasks. A bug is a bit like a puzzle or a riddle (except that nobody deliberately set it up for you to solve) and by demonstrating in a job interview how you systematically approach such a problem, the theory is that you're demonstrating what a good problem-solver you are in general. Even when the solver says things like "it's basically lanterns-and-bridge but with pigs" they're showing how they can recognize one problem in another and apply previous knowledge. That seems like it would be all good. People who can see patterns (all these numbers are one more than a power of two, or only letters with even ASCII codes are coming out wrong) do tend to solve bugs faster than people who don't. People who can quickly generate a hypothesis and test it are also often good debuggers.

The biggest problem is the false positives. People who aren't able to reason logically, generate hypotheses, spot patterns, figure out the weak link, make a leap, or even do basic math can nonetheless memorize, and these are the absolutely worst people to hire for software development or other creative jobs. Some of them can learn how to pretend they are working out the solution on the spot, and fool you.

A smaller problem, but still real, is false negatives. People who are excellent debuggers but just think it's silly that a group of such a size would not have enough light left for people to cross the bridge however they want, or feel that once one person is over the bridge the others can cross in the dark, and they just won't let go and get playful and try to solve the underlying math problem (arrange these numbers into pairs using the following constraints.) People who write great code and are offended by being asked to do something they're not good at in order to get a coding job. That sort of thing. There's also a real risk that the solver produces an answer that is better than the asker had in mind, and is rejected for not "getting it right."

I ask people if they like puzzles, and why. But I don't ask them to do a puzzle. And I would not study puzzles as part of a job-interview prep process.

Answer 2

To what extent is this practice effective or ineffective? Is there any correlation between puzzle-solving performance and job effectiveness?

If there is evidence of correlation, then for what types of roles? And what types of puzzles?

And if (as I currently understand to be the case) there is little or no evidence to support the effectiveness of puzzle-based job interviews, then why do companies persist with this practice?

Interviewers will often tell you that they use these questions to "see how you think", or "see how you think out of the box", or "see how you think under pressure" or some variant of that.

I don't believe any evidence exists to support such a claim. And I know of no company that can say "we have found that better puzzlers make better employees" with a straight face (puzzle-producing companies excepted).

Puzzle questions are a fad whose time has come and gone. At one time, companies like Microsoft used to ask questions like "why is a manhole cover round?" or "how many ping pong balls would it take to fill a jetliner?". Since folks equated Microsoft with "smart" and everyone wants to seem smart, they often chose similar questioning methods.

I once had a Developer ask a QA candidate I was recruiting "How would you find a needle in a haystack?". When I questioned the Developer if he thought that was a useful QA question, he said it would "tell him how the candidate thinks". When I pressed further, he said it was asked of him once in an interview and he thought it was a "good question". I never asked him to help with interviews again.

Apparently, Google no longer believes such questions are useful, and has some idea of the real purpose of these sorts of questions: http://www.nytimes.com/2013/06/20/business/in-head-hunting-big-data-may-not-be-such-a-big-deal.html?pagewanted=all&_r=0

According to Laszlo Bock, senior vice president for people operations at Google: "They don’t predict anything. They serve primarily to make the interviewer feel smart."

I agree with Laszlo.

Hiring and interviewing are hard. Coming up with relevant questions to discern the fit of a potential employee is hard. Asking puzzle questions is just lazy, in my opinion, and wastes valuable interview time. Unfortunately, most folks aren't trained on how to conduct an effective interview, so they just go with what they have heard is "smart" or "a best practice" and don't put any more thought into it.

After you are hired and advance in your career such that you are the hiring manager - choose better interview questions! No more "What is the air-speed velocity of an unladen swallow?" please.

Answer 3

Programming puzzles wind up, at best, being the equivalent of compiler micro-benchmarks. They're much more likely to tell you whether the programmer happens to have previously solved that problem or a similar one than to tell you how good they are at problem solving generally.

Answer 4

I'm gonna say no.

Puzzles are specifically designed to have a "best" solution and oftentimes, especially in programming jobs, you run into problems where there IS no solution.

Using a puzzle in the job interview process is more than a little condescending, especially since you're talking about someone else's career or potential career and certainly their livelihood, and yet you're treating it like a game. It's very, very disrespectful.

That being said, sometimes it's not the go-to thing for businesses but rather just a "best option" for places hiring. A LOT of managers, HR reps, CEO's, and sometimes even IT leads have no clue what the actual programmers do (and worse, they think they do) and they use these puzzles to put someone in the state of mind they think you'll be in while you're programming. It's less of a skills test and more of an "attitude under stress" test.

But you'll find that most people who hire employees based off of the results of the puzzle part of the test are going to hire terrible people. (You should focus on the resume!)

Answer 5

I work as a senior software engineer in Germany. In the last 6 years, I had at least 25 interviews. I was asked only once a puzzle question. Needless to say, whole interview was quite odd.

My experience is that people do not ask puzzle questions anymore. At least not in IT sector. For example, in google “What works well are structured behavioral interviews, where you have a consistent rubric for how you assess people”.

Even Microsoft, where the puzzle questions were born, stopped asking such questions since they realized it is a waste of time. There is this famous article "What would Feynman do?", which discredit such questions in quite a funny way.

Answer 6

I sometimes (but rarely) ask puzzles. When I do, the puzzles I ask are not ones that require an a-ha moment (a flash of insight) but are ones that can be solved with logical deduction: can the candidate break down the problem into a smaller problem, solve that, and then use that result to solve a larger problem?

Or sometimes I'll ask non-puzzle questions, and if I have time leftover, I sometimes might chew up some time with puzzles. But when I do, the point isn't to try to use it as some proxy for gauging their programming ability; the point is to find a data point for culture fit because programmers I work with tend to like puzzles too. And maybe the candidate has heard the problem before, but that can happen with non-puzzle programming questions too. If they have (and say so), maybe the information I gather from it is that the candidate likes puzzles, knows how to get answers, and is honest.

I think that they can be sometimes useful, but you have to be careful with what puzzles you use and think about what you're actually trying to measure.

Answer 7

Besides software development, one sector where puzzles / brainteasers / quantitative problem-solving is tested as part of the job interview process is in technical roles in finance.

I've heard a couple reasons given for testing interviewees in such a manner:

1) it's a form of stress test. Since there are a lot of jobs where a person has to be very quick on their feet and be able to apply their quantitative skills at a moment's notice, this should be tested somehow.

2) It's an efficient test of basic skills. Finance tends to attract people of diverse academic backgrounds. How do you verify that someone that says they are "good at probability" is really so? Does having a math PhD suffice? There are plenty of people with only basic undergrad math that have better probabilistic reasoning skills than many math PhDs.

Answer 8

I would say that it depends on the field. In some cases, like programming, candidates (generally) have a degree and some work history or at least school projects. They can be asked specific questions around programming so, to me, it makes less sense to ask them puzzle questions.

My specific experience is around Software QA. When I interviewed at Microsoft years ago for a QA position, I was straight out of graduate school and had no relevant work experience. There were no degrees in Software QA. So what should they ask? People that haven't done QA (and some that have) generally don't understand QA so it's hard to ask non-testers tester questions and expect to get good answers.

They asked me a lot of puzzle questions because they wanted to see how I went about solving puzzles. It wasn't so much about the answer, it was the thought process behind it. They also asked me simple testing questions... how would you test a date field? As I progressed through the interviews, the next person would ask questions based on what I was previously asked to see if I could learn and apply that knowledge.

You can learn a lot about a person by how they solve a problem. Do they give up immediately when they can't figure it out? Do they ask me lots of questions to try to gain more insight into how to solve it? Do they give one answer and get stuck? These skills are relevant to many jobs, not just QA.

I think the new hotness (well, not so new but anyway...) in interviews is the experience based question. Tell me about a time when you did X. The thinking is that the way you approached an issue in the past is likely the way you will approach it the next time it comes up. That's great... but how do you ask questions like those for a field that the candidate has no experience in? Not saying it can't be done but it's much harder... but, good interview questions are not easy to come up with.

Answer 9

All scientific evidence I have seen fails to demonstrate puzzles as effective. However, I believe they have some potential in non-scientific approaches. Consider that everything in the interview process is a tool. They're tools to help you and the company determine if they are good matches for each other (mostly to help the company, but it goes both ways). Accordingly, the interviewer uses any tools at their disposal. Some interviewers may find that asking a puzzle question is an effective tool in their toolbox to learn more about you, and then ask more pertinent questions. In such a scenario, I think puzzles are very useful. Its not the answer to the puzzle that matters, per se, but all of the subtle details surrounding it

I would like to switch over to fiction for a moment, specifically Frank Herbert's Dune. Fiction is often in a good position to explore the extreme version of a situation, and sometimes that can make the less extreme version more clear. In Dune, one of the groups has something known as the "pain box," or "agony box." You put your hand in it, and it uses nerve induction to generate the most excruciating pain in the universe. It is part of a test to determine if you are sufficiently human to go forward with their "interview." It is explained that it is not the pain that matters, it is how the interviewer observes your reactions, and interacts with you while you feel your hand melting off. In the hands of someone who does not know how to use it, the box is useless as anything besides a torture device. However, in the hands of someone who knows how to use it to accomplish their goals, it peels away layers of defenses so that they can conduct the real portion of the interview without interference from shallow things like memorization or bravado.

Now, after describing that interview, aren't you glad that they're really just asking you to solve a few puzzles?

Answer 10

There is a very interesting article about this topic on The Daily WTF. The main problem is that although lateral thinking puzzles are meant to measure the way you arrive to the solution, most of the interviewers will expect the exact same answers they have on their scoring sheets. So a candidate who thinks well and has a lot of good ideas, might be dropped in favor of someone who happens to have heared this puzzle before.

[the one who solves the puzzle in a brilliant and innovative way] will not get the job. Despite the complete absurdity of the design request, and the complete practicality of his answer, the job will go to a candidate who manages to answer the question by designing an extremely overcomplicated solution for a completely non-existent problem. And that candidate will be the same person who designs their software.