140

As a recruiter, I find it increasingly hard to recruit good software developers, due to the fact that it has become very easy to simply study for a job interview by reading any of those 14020 job interview questions for the software developer! books that are immensely popular these days.

It's gotten to the point where you pretty much can't ask a question without the applicant already having pre-studied at least a version of that same task and hence knows, roughly, what the solution is.

Now, this would of course not be a problem if knowledge of those problems is what we are looking for ... except, that isn't what we are looking for. Nobody cares if you know how to apply quicksort to some contrived, made-up example. The point isn't that you know how to solve those problems, the point is that you can figure it out. It's problem-solvers that we want, not problem-memorizers.

Because, once you actually get the job, you'll most likely never run into any of those problems from the job interview book again, so it's not your ability to memorize their solutions that matters; it's your ability to come up with a solution on the spot that is valuable. And yet, that is no longer testable, due to all these memorizers that read interview questions over and over.

How do we fix this issue? Because I am frankly tired of hiring promising software developers who answer each question perfectly in job interview questions, and then when you actually see them code in a real-world situation, you realize how little they actually know.

  • 2
    Comments are not for extended discussion; this conversation has been moved to chat. – Jane S Dec 30 '18 at 4:26

23 Answers 23

228

Provide the candidate with a genuine programming test in the interview - a laptop hooked up to a projector with a broken project, with a mixture of basic to complex bugs, and ask them to add some functionality, again ranging from basic to moderately complex. This is something nobody can really study for and will actually provide a showcase of their skills. Have technical people there talking to the candidate to get a feel for their personality and how they approach the problems faced.

For the job interview for my current role, I had spent 2 weeks studying programming principles, design patterns, database stuff etc. I was able to answer all the theoretical questions well and think I came across well

I was then handed a laptop which was connected up to the projector and shown a solution which contained a broken website with a defined number of errors which I had to debug and add some simple functionality. While I wasn't very familiar with the front-end framework used by the company, I was able to fix about half the bugs and then give my best guess as to what the causes were for the remaining bugs based on what I could see, and this was good enough to get me in the door.

The whole thing, between the theoretical questions and the website debugging session lasted about 2 hours.

  • 128
    Note that this approach, while highly informative, requires a technically competent observer to rate the interviewee's work. – chrylis -on strike- Dec 27 '18 at 22:50
  • 29
    @chrylis only a unicorn HR person would have the technical know-how to hire somebody without the need to having a technical person on hand. The issue is that the OP is saying that the technical tests questions they are asking have already been prepared for. – user1666620 Dec 27 '18 at 23:05
  • 39
    @chrylis in that case they're morons and deserve every incompetent they get – user1666620 Dec 27 '18 at 23:11
  • 68
    @chrylis Any approach will require a technically competent observer. There is simply no way in general for a non-technical person to judge programming ability, as far as I've been able to tell. – David Thornley Dec 27 '18 at 23:25
  • 122
    Be very obvious in the source code that this is all contrived source. You don’t want someone thinking you’re trying to get two hours of free labour out of them. – Ian MacDonald Dec 28 '18 at 1:13
140

once you actually get the job, you'll most likely never run into any of those problems from the job interview book again

If their job doesn't involve solving those kinds of problems, why test them on them in the first place?

Why not give them a real problem they would encounter on the job instead?

That's my recommendation. Surely you will have many examples of real on-the-job problems to choose from to find a suitable one.

  • 2
    This answer does not appear to differ significantly from user1666620's. – jpmc26 Dec 28 '18 at 5:13
  • 6
    They are somewhat similar in suggested solution. I thought it was important to address why their current method isn't testing what they actually wanted though, and suggest that new tests could be as simple as using recent challenges programmers have faced, rather than inventing new contrived tests. – Alexander O'Mara Dec 28 '18 at 5:28
  • 6
    @jpmc26 I would say that this answer more clearly identifies the root of the problem. The OP is clearly asking the wrong questions. Sure, the final suggestion is more or less the same for both of these answers, but stopping to ask the OP "why are you asking your current questions?" is a very important addition. – Conor Mancone Dec 28 '18 at 14:36
  • Agree, OP also wrote: Now, this would of course not be a problem if knowledge of those problems is what we are looking for ... except, that isn't what we are looking for. Which even further proves the questions they ask are completely pointless – Matsemann Jan 2 at 12:07
49

About a year ago I started to have discussions with candidates. There are many topics that are controversial in the industry and which do not have only one correct answer. Answers to questions about these topics usually start with "It depends…" – for example:

  • What framework is the best?
  • What database do you prefer and why?
  • Is a certain design pattern actually useful?
  • Focus on new features or fix bugs first?

First I ask the candidates about their opinion. Once they got their point across I simply pick the opposite and argue against it. Or I give an example in which their choice would not be a good fit answer and ask them if and how they want to change their answer with this new example/requirement? The following discussion will tell me a lot.

  • Did they only memorize some facts for the interview or do they actually have experience with this area and good examples to prove their point? Actually, I was very surprised by how often their argument is: because I always did it that way
  • How well are they able to bring their arguments and the complex technical details across? Are they good in teaching?
  • How do they handle resistance? Do they try to convince, do they try to please or do they start to be aggressive or arrogant? Are they open to learning?

I am aware that this might be very stressful for the candidate. Therefore the has to be done very carefully.

  • 13
    Just to add a consideration: Even leaving aside the stress aspect, if I got a question like that about something where I actually knew there was a correct answer for a given domain, and the interviewer confidently argued that my answer was wrong, I would have a rather dim view of the company's caliber, unless it was clearly evident that the interviewer didn't actually believe what they were arguing. – HammerN'Songs Dec 29 '18 at 4:48
  • 9
    @Hammer the questions that are posed here are specifically questions that many people have differing (and valid) opinions about. Even within a specific domain, there's no such thing as "the universally accepted best framework". And I would argue that, if someone can't handle a little stress in an interview, that might be an indication of how they're going to handle stress in daily work problems. – Kristen Hammack Dec 29 '18 at 14:48
  • 1
    @HammerN'Songs it should be possible to phrase the counterargument in such a way that as an interviewer i don't come across as "i know better", because reality is that i can't know better unless i have actual practical experience in that particular case, in which case what you knew to be correct, maybe wasn't that correct after all. – eMBee Dec 30 '18 at 5:17
  • 9
    @KristenHammack I don't think it's fair to judge someones ability to handle stress in the workplace based on interviews. I know plenty of people who are a mess during interviews but when stuff hits the fan during actual work they aren't even phased and just fix it. They're very different 'stress sources' and can't be used to make predictions about each other. – Onyz Dec 31 '18 at 13:33
  • 3
    @various - I think you misunderstood spickermann. It's not about arguing a ridiculous position you don't believe yourself. It's more like "Ok, so you think framework A is best. I've heard that people say great things about framework B, especially in (pick one thing where B is better than A). What do you say to that?" -- no prepared answer from a book will get him out of that, but actual knowledge in the field easily will. – Tom Jan 2 at 11:06
34
  • Have multiple interviewers. Your strongest engineers on the team should participate in the interview process - they definitely want to work with people better than the ones that are struggling, right? Let other engineers conduct their own 1 on 1 interview. And have another engineer do the initial phone screens.

  • Ask an open ended design question. for a problem of medium complexity. Ask the candidate to write out the header file, class declarations, box diagrams, etc... Does the candidate ask clarifying questions? After he's done with a basic design, introduce a new requirement that would break his design. Is he able to pivot? Pick one part of his design and probe deeper.

  • Ask a question that probes if they have a deep understanding of the platform fundamentals. Some examples I've asked:

    • "How does the C++ compiler implement virtual methods?"
    • "How do closures work in JavaScript?"
    • "Explain when you would ever want to use UDP instead of TCP."
    • "After you type www.foo.com into a browser address bar, tell me about all the network activity and protocols involved to get the content on the screen."
  • Probe for genuine interest in programming. This applies more for university candidates and junior engineers and less for experienced hires. An example questions is "Tell me about a program you wrote for fun that wasn't for work or school." A good candidate will tell you about an app, website, or hack that he did for his own personal pleasure. A weaker candidate will only tell you about the program he wrote to figure something out for work/school.

  • Comments are not for extended discussion; this conversation has been moved to chat. – Jane S Dec 31 '18 at 23:30
24

I've been most of my life as an electrical engineer, and back in the 90s many of the companies I'd interview with stopped asking me questions about electrical engineering. Between my schooling and my work resume, it was obvious to them that I knew enough about electrical engineering to meet their needs. What did they ask about?

My personality. My ability to work in (and lead) a team. My cross-discipline skills. My capacity to learn.

To be ruthlessly blunt, engineering skills (especially programming skills) are a dime a dozen. There are so many basically competent people out there that there is nothing about programming or engineering that's worth asking (unless you're interviewing a new graduate with no internship experience).

What's important is being sure the candidate is suitable for your environment. How well will they work with the existing personalities in your company? Do they bring more to the table than the skills needed for the basic job? How quickly can they come up to speed with changes in company direction? technology? or design standards? Are they willing to work beyond the basic job, providing (e.g.) scholarly articles, conference presentaions, patents, and other "bring honor to the company" activities? And can they show that they can do any of this?

I can sum this up with a comment I made to Philips Semiconductor recruiters long ago (as an employee). I can teach a 10-year-old to connect the dots when it comes to microelectronic design. Teaching them WHY you connect those dots is another matter. Therefore, you can't judge a new employee by how well they connect the dots. You have to find ways to discover how well they know WHY they're doing what they're doing. That's where teamwork and extra-curricular activities come in. They show depth of understanding.

  • 44
    "There are so many basically competent people out there that there is nothing about programming or engineering that's worth asking (unless you're interviewing a new graduate with no internship experience)." I don't agree. I've had a bunch of people in for interview who looked great on paper but couldn't answer even some pretty basic questions about fundamental commonplace technologies. We didn't even get as far as fit, though that is of course also important. I'm happy to hear that you were qualified for the roles you applied for, but this is definitely not the case for all candidates. – Lightness Races with Monica Dec 28 '18 at 1:00
  • 6
    Right, this is not the case with software. Almost all "programmers" are barely competent, have a self-taught hobbyist vibe, and can't actually do anything. The OP is perfectly correct that folks drill to answer "interview type questions" you can look up online. The QA is about how to solve this problem. – Fattie Dec 28 '18 at 7:19
  • 4
    I think part of the disconnect here is that there is a standard, codified body of knowledge that more-or-less an EE makes. Not so for programming. There are multiple problems stemming from that lack of a base: two candidates identical on paper might be orders-of-magnitude appart in actual ability, a candidate might be a true genius in one discipline but totally ignorant in others that might be crucial to a particular job, etc. You can't even use compsci as a base: there are plenty of competent front end people with little understanding of e.g. data structures or how compilers work. – Jared Smith Dec 28 '18 at 15:02
  • 4
    @JaredSmith Oh no, EEs are at least as variable as programmers, the number of candidates who cannot tell me how resistor value impacts noise performance, or why you fit decoupling caps is scary, and don't get me started on the weirdness you sometimes hear if you ask a transmission line question. A drawing of a power supply, a resistor and a long cable plus a 'scope gets you really weird answers. All of these are the equivalent of 'write a bubble sort' in the EE domain. – Dan Mills Dec 28 '18 at 15:39
  • 5
    The title of the question specifies "junior developers", so they're not talking about people who already have a resume. – Kristen Hammack Dec 29 '18 at 14:40
19

The fundamental problem here is that you are not using coding problems correctly as an interviewing technique. It sounds like your interviewing technique looks like this:

  • Pose a problem
  • Did the candidate successfully solve the problem? HIRE. If not, NO HIRE

This interviewing technique is, as you've discovered, (1) very easy to "game" by the candidates, and (2) gives you low quality "signal" about their actual abilities that are relevant to the job.

Here's how I give a coding problem interview:

  • Pose a problem. The problem should be one that could be easily solved by anyone in this job, it should be relevant to the job, and it should not rely on any "culture-specific" knowledge. In particular it must not require an "aha-moment" insight.

For example, when interviewing for a position where the candidate would be writing a standard library for a programming language, I'd ask how to implement one of the easier methods in that library. When the position is designing tools that spot defects, I'd put some legal but buggy code up and ask the candidate to find defects and describe why they are defects and what the severity is. When the position involves manipulating parse trees in a compiler, I'd ask about extracting facts about trees, like whether a balance property is violated. And so on.

  • Make sure the candidate understands the problem by working some simple examples.

These simple examples are test cases; the candidate should recognize that when it comes time for them to justify their solution.

  • Give the candidate an opportunity to ask clarifying questions.

This is particularly important if the problem is in any way underspecified. "Do we have to take daylight savings time into account in this date computation?" "Do we know for certain that integer arithmetic is twos complement 32 bit?" "Is the tree shallow enough that we can write a non-tail-recursive algorithm?" and so on.

This is valuable signal about the candidate's ability to analyze a problem, communicate about it, and resolve an ambiguous situation. Those are all valuable job skills. Use this signal.

  • Once the candidate can begin to solve the problem, how do they begin? Do they use "test driven" style, where they write some test cases first? Do they write a clear function header that describes the desired inputs and outputs? Do they follow standard practices for naming? If using an OO language, does the candidate follow good OO design principles? In short, can they use the function abstraction effectively?

  • If the function has error cases that are not caught by the type system, does the candidate use validity checks, assertions, or any other technique to ensure that preconditions are met?

  • Are there "easy out" cases that can be handled after error checks? Does the candidate write code for the easy outs? Can they describe why they made that choice? Easy out cases give us a tradeoff between code complexity and performance; under what circumstances is that tradeoff justified?

  • Once the entire method is written, is the method written a valid, correct implementation of the spec? Can the candidate convince you of this? If the code is not correct, then do they try hard to convince you that it is correct? Can they use test cases that they've already got to provide evidence of its correctness? Can they come up with new test cases? Are there assertions that verify postconditions? Work with the candidate until you are both convinced that the code is correct because it is correct.

  • Now we have a correct implementation. We are not even close to done yet. The problem should have been so easy that coming up with a correct implementation takes nowhere close to all of the interview. Next question: analyze the performance characteristics of the algorithm you just wrote. What resources does the candidate identify? Do they understand that performance is about user requirements, not about raw speed? Can they make tradeoffs between space and time? Do they understand amortized performance vs worst case performance? Do they understand throughput vs time to first result? Can they describe to you exogenous effects on performance like collection pressure in gc'd languages? Can they correctly give the asymptotic order? What modifications might they make to the code to adjust its performance characteristics?

  • Now we see how deep the candidate's toolbox is. Tweak the problem statement to make it harder. "What if these two dates could be in different time zones?" "What if we ran this code on a machine where pointers could be of different sizes?" "What if we knew the tree was likely to be highly imbalanced on the left, and we could not do any left recursion?" "What if we're in a memory-constrained environment and we cannot allocate arbitrarily many objects?" and so on. All of the above are situations I've encountered at work. Take examples from real-world situations you've had to deal with. Get good signal from how the candidate deals with them.

In short, I don't care if the candidate has seen my problem before because solving the problem is the least interesting thing they are going to do in the interview. I am not looking for ability to produce a standard solution to a standard problem; I assume I'm going to get that, and if I don't, it's an easy "no hire". I am looking for ability to design, specify, clarify, implement, test, justify, analyze and extend solutions.

  • So, how many buggy variations of strncmp() (or its equivalent in some other language, or some similarly, reasonably simple, standard library function) have you seen? :-) – a CVn Dec 29 '18 at 20:21
  • 2
    @aCVn: Rather a lot. When I was working on standard library code I would ask how to implement a simplified version of the "how many days are between these two dates?" function. It turns out that a number of candidates do not actually know how subtraction works, and those got no-hired. – Eric Lippert Dec 29 '18 at 20:53
  • 2
    What's between the lines of this answer is that it should take at least as much interviewer effort to pose the problem and evaluate the answer as it does for the candidate to solve it. And +1 for the last sentence. – Blrfl Dec 31 '18 at 13:05
  • @Blrfl that's the real takeaway. This question reads to me closer to "Why am I not getting quality candidates using a canned screening process"... because you're using a canned screening process. – TemporalWolf Dec 31 '18 at 23:05
  • 1
    @Piskvor: That's exactly right. However there are some subtleties. I don't want to have an interview that is a test of trivia, or that gives a disadvantage to people from different cultures. Everyone lives in a world where there are time zones and daylight savings time, so I'd expect people to be able to criticize a date format on that basis. But not knowing why Ukrainians celebrate Christmas on a different day is not something I want to ding people for in an interview! – Eric Lippert Jan 2 at 16:11
10

Temporary paid trials.

Welcome new user, I've come to believe that the only way to find software engineers is, once you've found someone who seems to know what they're talking about in the specific technology at hand,

  • Simply pay them for a one or two week freelance contract. Actually have them jump in and do a real project on your real product with the real team.

This is becoming more and more common.

I think that's it.

It's the only way to know.

Everything else is useless, as the OP points out.

The amount of time/money you'll spend on the sundry alternatives ... might as well just put them on the project for a week.

(Depending on your style and project, you might retain 'em for a time period (a week) or some "task" for $x. Either way.)

If you consistently take this approach, you'll find after a year or two that, the money you wasted doing this approach, was indeed far less than the sundry other costs of search and hiring.

"Because I am frankly tired of hiring promising software developers who answer each question perfectly in job interview questions, and then when you actually see them code in a real-world situation, you realize how little they actually know."

Temporary paid trials.


Some further thoughts, the tenor of this QA is

• For software engineers, recruitment processes fail - indeed they're often "hopeless". (One apex of this, and the particular one raised in frustration by the OP, is that hopeful programmers now just "game" the "technical questions" phase of things.)

• Note that even "google-style" absurdly detailed/extended recruitment processes ............ often completely fail. Programmers (in particular) often "just don't work out" no matter how careful the recruitment processes.

• By all means in any industry or mode of occupation, folks sometimes "don't work out" even after careful recruitment. However, this is especially a problem for software engineers - it is notorious.

{You may ask - why? How come it's particularly a problem with programmers? I know precisely why this is, and some people have their own theories on it, but no need to start another argument!}


Even more thoughts!

This answer has a big pile of downvotes and a big pile of upvotes.

Apparently

  • for many folks this is a totally obvious idea

  • but for other folks it is freaky crazy stuff.

After all, setting aside just programming per se, back in the 90s one of the keys to the staggering success of General Electric at the time was Jack Welch's scheme where each manager simply fired the bottom 10% of people each year! (His book is terrific BTW.)

And I cringe to type the phrase "the gig economy" but in "the gig economy" we are all on "a trial basis" - from day to day and forever.

Winnowing programmers is incredibly, notoriously, difficult.

  • 1
    Comments are not for extended discussion; this conversation has been moved to chat. – Jane S Dec 29 '18 at 6:13
9

Use Whiteboard interviews, and ask problems which are related to the skills and business domain that you're interested in. Look for people that you can work with, even if they aren't perfect matches for the position, because you can always train the right person if they have potential. Ask existing employees for recommendations (and offer a finders-fee)

  • 9
    Pete, I fear that OP is indeed talking about whiteboard-type problems! – Fattie Dec 27 '18 at 18:49
  • 6
    @Fattie - Yep - but a good thing with whiteboard interviews rather than take-away coding tests is that you can change the parameters halfway through the question to throw a spanner in the works – PeteCon Dec 27 '18 at 18:52
  • 18
    Don't ask people to code on a whiteboard! That's cruel! – Lightness Races with Monica Dec 28 '18 at 0:56
  • 13
    These whiteboards interview are exactly the thing that got lots of books printed to go through them. As a developer, I like spending my time into actually learning my job and executing it, but my job is not writing programs on a whiteboard. – Pac0 Dec 28 '18 at 1:43
  • 3
    @Abigail What I mean is that, and of course I can only speak for myself but surely this can't be uncommon, I hate writing code on the whiteboard because it's really uncomfortable and there's no way my hand can keep up with my thoughts. Give me a laptop and a projector already! Watch my fingers fly. – Lightness Races with Monica Dec 28 '18 at 13:45
7

After getting through enough basic questions to show that they have some minimal (by my needs) ability to write code, I move on to the tougher part of the interview, based on two subject areas, their work and my work.

Ask for information about a recent project at their current job. Even with respect to NDAs and IP restrictions, they should be able to explain the business problem, the complexity of the problems and how they overcame them. If they can't explain their own work to me so that I can understand and evaluate, then I worry about how successful they will be.

Ask them to white board a design/solution to one of your current problems. The expectation should not be a working solution but how they approach the problem. Provide enough information that they have a high level scope for the problem. They should ask good questions, and you should provide good answers so they can proceed. They should be able to sketch out some sort of solution. In the end their solution may not work out, but you will be able to see how they work on a problem and whether or not they should be able to work on the projects you will have.

6

Simply. You just grill the applicants on the implementation details and change requirements on them to see how they respond. Make them use what they've memorized as building blocks.

To be frank, problem memorizing isn't an issue. Memorization is actually one of the key techniques chess players use to learn the game. By memorizing algorithms your applicants are developing a library of solutions from which to work from. This means that they'll be able to solve problems better in the future, if the patterns and implementation of the patterns they're learning are actually useful.

What you need to test is that last part. By varying the problem in real time you get to see how well the applicant actually understands how to use what they've memorized. This means that you can ask them about better ways to solve the question and you can even ask them to compose their solution into a work flow to do something completely different.

What you want is someone who can solve problems. What you don't want is someone who solves problems the way you want to see them solved. If everyone's memorizing solutions, all you need to do is test their ability to use what they've memorized. When it comes down to it, everyone uses what they know to break down what they don't know. If your devs can use patterns they've already mastered to break down problems of increasing complexity, then they would have been able to reverse a linked list if they've never seen anyone do it before.

As a recruiter, this is as far as you can reasonably be expected to go without trying to get free labor out of your applicants.

  • It's funny you mention reversing a linked list. I expect that's the type of question OP is complaining about. Personally, I go even more basic than that. I ask what data structures they have used in their favorite language, e.g. in Python, there is a list and a deque. I would ask about differing performance characteristics between them for operations . They may have this memorized but not understand that a list is backed by an array whereas a deque is backed by a linked list. So despite their ability to reverse a linked list, they don't really understand why you use linked lists. – Chan-Ho Suh Dec 28 '18 at 0:37
  • 2
    @Chan-HoSuh that would get me for sure. I've been writing scripts for work for a while now and I've never had to use a linked list for anything. I honestly struggle to see the point, especially in python. If I were asked to reverse one in an interview, I'd probably tell them they're using the wrong data structure to begin with. Would not be passing that one hahaha – user53651 Dec 28 '18 at 16:16
  • 1
    Actually I started thinking the other day, why don't we implement a deque with a backing array instead of a linked list? So some interesting thoughts have resulted from this side thread :) – Chan-Ho Suh Dec 28 '18 at 16:21
  • @Chan-HoSuh The point of using a higher language is to abstract away the nitty-gritty. It's like asking a truck driver how a carburetor works... sure some of them will know and may be able to eek a little extra performance, but you won't learn much about how they drive. If you have to look for the man behind the curtain you're probably using the wrong tool. – TemporalWolf Dec 31 '18 at 23:01
  • @TemporalWolf sounds good in theory, not so much in practice. What do you do with someone who, in Python, insists on prepending to a list in a loop, creating an O(n^2) algorithm instead of an O(n) one? Turns out you do need to know something about how a list in Python is implemented to not do stupid stuff. And as you learn more Python, you discover more of these things. – Chan-Ho Suh Dec 31 '18 at 23:10
3

In some ways, studying for the interview is a good thing. You probably already ask that people study about four years in a University to prepare for the interview.

Now, if your interviewing process is scripted, and the script lacks any kind of variation, then it is just a matter of time before the answers become memorized in the population. Solutions include:

  1. As questions that are generated, with variable fields that require the work to be done specific to this asking of the question.
  2. Change the question set from time to time, to minimize the time to build the pool of answers.
  3. Change the entire approach, giving them a trivial project with specifically added bugs, asking them to fix it and add a known feature.

Also, consider the problem in the larger sense, if you are asking for information in an interview, and the information was memorized, then that should be an ideal candidate, unless you are asking for the wrong information.

You are deciding that the information isn't what is wanted, so you might be able to fix the problem by asking for the right kind of information.

3

Being able to come up with a solution on the spot isn't really that great a qualification, either, at least IMHO. I've certainly come up with a lot of on-the-spot solutions that in retrospect weren't all that great, and have sometimes spent weeks or months finding a decent solution to a really hard problem. (And sometimes the solution is just remembering that I saw something similar in a code example somewhere...)

You don't say whether you're trying to hire new grads or experienced developers, but in either case I think you really have to find some way to look at what they've done previously. Sometimes it's looking at published papers (even if the person isn't the principal author), sometimes it's a personal recommendation, sometimes just asking them to show you something they've done that they're proud of. FWIW, I don't think I've ever gotten a job that I interviewed for that didn't have at least one of those things going for it.

3

I believe you misunderstand the reason why big companies ask algorithm questions. Writing out the algorithm for quicksort is probably not a very useful skill for the majority of developers. Likewise figuring out the best way to parse a 1 million character string is not something people routinely do at work. However knowing how to solve these problems is a great way of filtering out the dumb candidates. If someone is too dumb for the job, they won't be able to coherently answer any of the interviewing puzzles commonly asked at job interviews. Even if they memorize all the answers, you could easily break their pattern by asking a slightly different variation of the question or adding a bonus question on the spot. In contrast, a good candidate would have no issues preparing for the "test" and would pass with flying colors.

Interviewing is about filtering out the bad candidates, rather than about finding the best candidates per se. There's no harm in having someone prepare in advance, as this wouldn't help someone who's not prepared to work on the right level.

  • This is quite true. "interview tech questions" are just a filter. (like having "a reasonably professional CV" is a filter.) – Fattie Dec 28 '18 at 7:25
  • 1
    I think in certain environments filtering out the "dumb" people is a good idea, but I think there are a number of "dumb" developers that are quite successful nonetheless at delivering quality code. – Chan-Ho Suh Dec 28 '18 at 16:17
3

The problem is that you're not testing for the skills you're looking for.

A process I've used both as an interviewer and interviewee that has worked well:

  1. A quick 15 minute phone screening to find out if the person is competent and someone you could see yourself hiring.

  2. A simple coding exercise for the applicant to complete on their own time. This is where you get a feel for actual problem solving and coding skills. The exercise should not take more than a few hours and you can optionally pay them for their time.

  3. The actual technical interview, which involves reviewing the coding exercise and discussing the solution and the decision making process.

Most of the (currently) highly rated answers to this question suggest some sort of live coding as a part of the interview. I would highly discourage this as it is not at all representative of a real world scenario. It's a high pressure, unnatural situation where a candidate is likely to produce an inferior solution than they otherwise would, with no additional benefit.

  • 1
    I, for one, absolutely agree with that -- I am the kind of person that gets incredibly nervous and literally get "dumb" when presented with an on-the-spot tech quiz/interview. Every other interview that I actually get a take-home exercise I generally ace it and pass them. All my people references and one-on-ones about my performance are positive, which I believe it is indicative that I am a good worker/developer. And in my humble opinion, that's what matters in the workplace rather than knowing all gotchas for a tech exercise that you don't really have enough time to explore. – Matheus Felipe Jan 3 at 22:10
2

As an applicant, there are a few things I noticed which can still be tested during an interview, even if you're not in the IT field yourself:

  • Behaviour-based questions

The objective is not to nail the person you have in front of you, but actually see what kind of person they might be. This gives a short window of whether or not they might be suited for working in the company.

  • Past experience questions

[Could you] Tell me about a situation when...

For this one, the objective is to invite the person into telling you more about their experience in a specific situation. This type of question can assert their skills as well as their self-being.

It can be useful if you know the IT teams has some situations which can be difficult to handle (having to handle a lot of tasks in short notice, having to handle tired clients, etc).

  • Problem-solving based question

You mentioned you are looking for problem-solver.

I once had an interview with a company where the interviewers didn't ask any programming based question. In order to assert my problem-solving skills, they instead told me about a situation, and I was told to think aloud so they can witness my problem-solving skills in action. For example:

We have a plane with 50 seats, and 50 passengers awaiting to come in. The first two passengers lost their cards, so they will sit in a random seat. The next passenger will sit in their seat if they can (if it's not already occupied); otherwise, they will sit in a random seat. What is the probability for the last passenger to sit in his own seat?

The objective is not for them to find the solution, but to show you how they tackle the problem.

~~~~~~~~~~

For each of these questions, remember to take note of the applicant's answer. This way, you can better discuss about them with the IT department manager.

2

I don't think you need to change the questions, just how and what you ask.

It's gotten to the point where you pretty much can't ask a question without the applicant already having pre-studied at least a version of that same task and hence knows, roughly, what the solution is.

This isn't a zero sum thing, you can feed off the solution for an entire interview on it's own if you want, and get a deep understanding of the interviewee's skills (assuming as the interviewer you have your own technical skills).

You present the question, the interviewee creates a solution (maybe code, maybe whiteboard). This is the start:

  • Take me through the way this works
  • What factors led you to choose this solution?
  • Are there any other ways of doing this?
  • Why is your solution better than the others?
  • Are there any potential advantages in the other solutions missing from your solution? what scenarios might that apply?
  • Having just written this off the cuff, any refactorings you'd like to do now you've looked it over?
  • If you didn't do this test driven, what tests would you be looking to write/do to ensure this is correct?
  • Any bugs you've noticed?
  • Do you think the code is production quality? Could another developer support/refactor it without knowledge transfer from you? What would you need to change to ensure it's intent is understood?

Someone who has memorized an algorithm or pattern without understanding it will not be able to drill down into detail, but a good developer should be able to have discussions on these topics, even if their raw solution wasn't perfect (and seeing someone discover a better solution while talking to you and be prepared to call it out shows a maturity over code/design reviews and refactoring which are highly desirable).

2

The best interview I ever gave was one of the simplest interviews I gave and was also the best my skills were ever measured. This is what its 5 rounds were:

  1. Telephonic with the recruiter: who asked me to verbally go about explaining the answer to a simple coding problem. (The recruiter was not technical.) Answering him demonstrated that I can explain my solution to non-technical people, proving that there is a real connect between the skills mentioned on my resume and practicality.

  2. On-site 1: One of the company's senior dev demonstrated their product to me. He paid attention to the questions I was asking during and after knowing their product. This helped them accurately gauge my interest in that domain and whether I was genuinely curious.

  3. On-site 2: A tech-manager from the company brought one of their own pieces of code and asked me what I think of it. The code had basic issues - like not using try-catch, not closing resources, poorly named classes and variable etc. The code was working, but was not written responsibly. The corrections I suggested, demonstrated my design skills. If one is not a responsible programmer, this interview would end up being the hardest. Especially for solution muggers.

  4. On-site 3: A senior developer discussed with me the design of Merge Sort and after correct design was reached, asked me to implement it in my language of choice. The design discussion was relaxing as I was sure that him and I were on the same page about what we were going to implement.

  5. On-site 4: Another senior dev asked me to explain one of my own projects to him. This demonstrated that I actually understood things being done in my current job instead of doing short term and nonsensical code patching.


All the rounds were very relevant and nowhere did I feel that I was being asked to do something out of the blue especially for this interview. Every task was gauging how good a software developer I was, rather than how many answers had I memorized.

Not surprisingly, this company was very highly rated on Glassdoor and had very low attrition. Employees didn't write much about what they were doing but they all were really invested in their company and their work.


In my job, the right solution to difficult problems has come after discussion between teammates. That is what a good interview would try to simulate. Just asking difficult problems in an interview is generally not helpful.

  • I'm not sure I'd call issues "like not using try-catch, not closing resources, poorly named classes and variable etc" design issues. In my book, those would be code quality issues. You can have completely squeaky-clean code that dots all the "i"s and crosses all the "t"s at the level of lines of code up to functions and classes, yet the overall system design can be a total mess which still makes the code brittle. For an example of a design issue (at least in my book), is all the presentation logic cleanly separated from all the business logic? Why would that be important or desireable? – a CVn Dec 29 '18 at 20:15
  • @aCVn: You are right. I mixed two thoughts while writing the answer. What I wanted to say was that the code review round tested my understanding of Code Quality as well as Software Design. – displayName Dec 29 '18 at 20:45
2

A number of people have answered "The questions don't match the needs of the job" - and that the solution is to change the questions to something that more accurately reflects what's needed to know how to do the job.

I'll take a different angle: This probably has a lot to do with recruiters/interviewers not putting in the time to create their own questions.

In our group, we're currently interviewing .NET devs, and here are some of the questions from our interviewing process:

  • We've got these two SQL tables with these columns. Write a query that gathers data X, filtering down based on Y, with a corner case of Z being handled in a specific way.
  • Here's a badly-written 8-line C# function; Code review it.
  • Write a C# function that takes an input of a filename, and outputs how many bytes there are before the first null byte in that file.

The answers to those probably aren't going to be found in a '140 questions to memorize before an interview' - because we came up with the questions from scratch. We didn't copy them from another company, or crib them off a 'Good Interview Questions' site. Yeah, it takes more work... but we don't have to worry about someone already having the answer memorized.

1

If your candidates are able to pass your interviews by studying, isn't that a strong indication of their ability to study and learn? If you're hiring fresh grads, that's really the most valuable skill to hire for.

If an experienced candidate is doing it, at the very minimum it means they're highly motivated to study for your interview. Which means they're strongly interested in coming to work for your company. And also, it's a strong signal that they haven't lost their ability to learn and grind after leaving college. You should still ask them questions about their work experience, achievements, and collaborations with co-workers to get a better sense of their overall ability.

It's problem-solvers that we want, not problem-memorizers.

Very few people can come up with quicksort, or Floyd's cycle detection algorithm, or finding the longest increasing subsequence in the timespan of an interview having never studied the solution. Those who can are unlikely to be interviewing at your company for regular dev jobs. No offense, I'm sure it's a great company, but statistically most of your candidates are not going to be of that caliber - because very few programmers in the world are.

On the other hand, if your candidates are able to apply solutions to problems they've seen before to variations of that problem or to related problems, it's a signal that they're good at pattern-matching and identifying what type of problem they're looking at. Conversely, if they've never seen that type of problem before you also have signal that they know how to research a solution and understand it - because after all, they studied for, and passed your interview.

  • 1
    The thing is, you shouldn't have a test that can be aced by memorizing what can be quickly looked up, because such memorization is not a useful job skill. Rather, you should have a test/question that requires the kinds of thinking necessary on the job that a search engine can't help with, or at least requires using it in a clever and inventive way. – Chris Stratton Dec 28 '18 at 5:29
  • It's not possible to memorize solutions to so many programming problems. Pattern-matching previously-seen solutions to new problems isn't the same thing as memorizing answers. If they're asking trivia (what are the arguments to method foo in library Bar), that's a different issue. – Jay Dec 29 '18 at 6:27
0

I've had to recruit 5 developers of different skill levels over the course of the past year and it's not been an easy process. The best method we found was to make use of online programming tests to filter out the weaker candidates before we interviewed.

We picked a coding test website (CodinGame, though other alternatives are available) and asked candidates to sit a test before we interviewed them.

Pros:

  • Filtered out the 'looks good on paper' candidates who didn't live up to their CV / resume.
  • Allowed us to see the candidate's working - we offered interviews to people who didn't score highly on the test but by reviewing what they had done to solve the problem gave us a good idea of their analytical skills.
  • We viewed candidates who weren't serious enough about the position to do a 50 minute online test as not worth interviewing, this filtered them out.
  • We could set tests at different skill levels.

Cons:

  • Some candidates, when faced with several interviews, would go for the path of least resistance and choose the the ones that didn't involve a coding test.
  • You need a few candidates to take the test to get a benchmark of what is a decent score.
  • There is a cost involved in taking the test.
  • Just a note on coding tests, they are very much about the individual. In reality, coding is a team sport. It would be much more revealing to bring a group of successful candidates into a team coding test to see how they work together. – Dominic Cerisano Dec 29 '18 at 20:45
  • @DominicCerisano - That's a good point. From my own experience, however, we struggled to get a lot of decent CVs in and recruited over several months. Wwe couldn't have realistically have got enough candidates in at the same time to do this. – GrandMasterFlush Dec 31 '18 at 9:09
  • If you are only interviewing seldomly, then consider involving some of your existing team in the test. Toss the candidate a nasty bug in a non-sensitive repository and chase it all the way to a merge. The others can be as helpful or as obstructive as they wish, kind of a Kobashi-Maru scenario. How they react to team dynamics is just as important as what they know. – Dominic Cerisano Mar 12 at 0:39
0

Flip it: Make it a test for gamers. Pick those who fail...

Anytime you have a metric that is being intensely gamed, the metric becomes useless.

But it's worse. Excellent performance on the metric becomes a flag for a gamer.

And that makes it better :)

So you flip it. You optimize this testing to identify people who have learned to test well. Don't make your testing gamer-resistant -- make it even more gamer-vulnerable, so gamers will do exceedingly well on it, and people who have not "trained up on interview books" will not. Also include other interviewing or testing that the interview-training books cannot prepare you for.

Now you are going to see a pattern/signal. You will see candidates who score really well on the gamer test, but flounder on your other stuff. Those are gamers, don't hire them. Conversely you see candidates who show relatively poorly in the gamer test, say in the 25-50th percentile compared to other candidates. they're not incompetent, they're just getting trounced by the gamers... and they do just fine on the other stuff. Hire them.

  • No, it's data science. Data does what it wants, not what you intend it to do... – Harper - Reinstate Monica Dec 31 '18 at 0:49
  • This is a very bold suggestion. Worth trying. – Jay Dec 31 '18 at 5:17
  • Trouble is, this idea is straight out of game theory, which a gamer might already be aware of. The Nash equilibrium would probably still come down in their favor (if they know that you know that they know, etc.) – Dominic Cerisano Mar 12 at 0:46
0

You say

"Now, this would of course not be a problem if knowledge of those problems is what we are looking for ... except, that isn't what we are looking for."... "It's problem-solvers that we want, not problem-memorizers." .

You just answered your own question. If you want to test their problem solving skills, just ask them to explain their earlier projects and work in detail. Ask them questions like why they chose a particular platform/programming language. Ask them what were the requirements and how they were implemented etc. Questions like these will give you an idea about how much complexity the candidate has handled before. Some have suggested to actually give them real debugging problems during the interview. Although a good approach, it can be time consuming. Also, every bug is different and even great programmers struggle if they don't know the history of the code. I think you can have a mixture of all three approaches i.e some textbook questions + earlier projects + actual bugs.

0

Instead of asking specific pre-prepared questions, ask the candidate to bring samples of their work. For graduates that could be stuff they did on their course or as a hobby.

You can then discuss it with them, ask them to explain what it does and any particularly interesting parts of it. You will get a lot more from this process than you would simply asking technical question. You can see how critical they are of their own work, and how they think about solving problems.

It also takes the pressure off them. If you ask them to do coding tasks under time pressure and with you watching they may not perform well, and it's a very artificial environment and something they are likely not used to.

You can use this discussion as a jumping off point for talking about things more specific to the job, but the focus with graduates tends to be on the ability to think and learn.

protected by Jane S Dec 28 '18 at 5:47

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.