How can I determine if a coding task is too hard for an interview candidate?

Question

I am a senior software engineer and a lead developer in a company with 250 employees. The company is hiring mid to senior level developers. I am doing technical interviews, even though I have been here for only 2 months. I don't have much prior experience doing interviews as my previous company was much smaller and hiring new backend engineers was very rare.

A part of the interviews is live coding, usually through Skype. The candidates can use their own IDE or use a collaborative real-time editor, such as CollabEdit. They have 20 to 30 minutes to complete the task. I do not care about the syntax, the point of the task is to prove the ability to find the solution and implement it.

A lot of promising candidates (having good experience, knowledge and ability to evaluate why a particular framework/technology is used) fail on what I think are easy coding tasks (for example, finding the second biggest number in an array). I understand that they are under pressure, but I try to guide them and make them feel comfortable. I usually also provide them with code skeletons to get them started.

I have consulted with the Technical Lead. He says if the candidate is unable to do this, we should not continue, as we are looking for programmers and we expect them to write code.

Since so many promising candidates are failing these coding tasks, I am wondering if they are really as "easy" as I had assumed. How can I determine if a coding task asked in the interview is too hard?

Answer 1

How can I determine if a coding task is too hard for an interview candidate?

Your question is ambiguous. Let me try to crisp it up. I think what you are really asking is:

We have a process for obtaining a signal from candidates about their fitness. If the process produces a "no hire" signal for candidates that are actually fit, then the technical question portion of the process might be too difficult. How can I determine if a question is too difficult?

That's easy; as others have said, put your coworkers who are known to be fit through the process; if the process consistently rejects them, then there's something wrong with the process.

But that's not the problem you have. This is a question-and-answer site, and you've asked the wrong question.

The problem you actually have is a significant fraction of candidates are plainly unfit, and this is wasting time, effort and money that could be better spent elsewhere.

You have a pipeline problem, not a coding question problem. Something in your candidate pipeline is producing a large fraction of plainly unfit candidates. As is well known, a great many programming job applicants have no ability whatsoever to write even simple programs: https://blog.codinghorror.com/why-cant-programmers-program/

Ask around and see if other interviewers are having similar problems. If they are, then push back on management and recruiting to see if they can figure out what is incentivizing recruiting to keep sending you unqualified candidates.

Answer 2

How can I determine if a coding task asked in the interview is too hard?

Seems that those failing candidates are evidence to support that your coding task may more challenging than you think.

Empirically, you can determine if future coding tasks are "too hard" if you see several candidates failing or taking too much time doing them.

You can also have yourself or some technical coworkers to try do the coding tasks, take your time doing it, and then compare your results with whatever your expectations of that particular tasks were. You can then determine if it needs some simplification to be able to complete in a regular coding interview.

Keep in mind that, usually, coding interviews or tests should not be too long. Otherwise you will not be actually evaluating the knowledge on the matter, but instead just seeing who is better in pretending to know a programming language (in 2 hours anyone can search and become an "expert" on any Stack Overflow tag). Usually, simple and straightforward questions or tasks are more effective in measuring the language knowledge as well as some creativity and improvisation.

I also suggest you take a look at this question, where several answer are given that could help decide if a specific task should be removed or not.

Answer 3

I had similar problems, with very simple coding questions. It seems that many people are what we termed “wizard's apprentices” after the IDE code-generator wizard and the Micky Mouse cartoon. They think they know C++ but really have no idea how to write a class, even a simple one.

Here is the first one. I always read it so as to give the exact same wording to everyone, but from memory now: “Write a class whose constructor takes two integral arguments and has members to return their sum and difference.”

I expected to see different levels of how well they scrutinized the specs to nail things down before starting, and people who used templates rather than choosing a specific type, etc.

What I got was a surprising number of people who could not write a class, period.

---

for example, finding the second biggest number in an array

As I said, nail things down before coding:

• Can I modify the array?
• Did you mean the second element in sorted order, or do you mean the second largest unique value ([1,2,2,3,3,3]is the answer 3 or 2?)?
• Should it deal with arrays that have fewer than 2 elements (or unique elements, depending on answer to prev)? Throw exception, plan for optional return value, or what?

Should I clarify the specifications (and get signed off on it) and then write critical test cases, as in "real life"?

Your stated goals are at odds with the unclear specification. As stated, you really should test engineering skills first, then coding skills. And do you really want to hire someone who will throw together an unreadable overly-clever un-maintainable mess? I question the wisdom of not caring about syntax.

If you just want to know if they can approach the problem, do orals rather than a coding task. Have him explain how he’d do it: e.g. I know about the standard algorithm, so duh. If that does what you really wanted. Be impressed that I have useful references bookmarked and always open, and don’t worry about the final step of writing a sample program that contains a line that calls this function.

So I question whether your test is really ready for someone to just code, or if the issues I pointed out would paralyze the candidate who is told that this is a complete specification ready to code?

Answer 4

How can I determine if a coding task asked in the interview is too hard?

There are a variety of ways. By far the easiest is to find a community of people you want to hire and ask them.

This might be a meetup type group. This might be internal people. Maybe online forums. For programmers you will get tons of feedback because most programmers can easily be nerd sniped with "hey, can you help me see how long this programming question takes?"

Figure out standard introduction to your questions, too, so everyone gets the same information. Maybe a talk sheet for the problem intro. Interviewing can be stressful for both parties.

Then, once you have a time for the task, multiply it by at least 1.5 to account for nervousness inherent in interviewing and bam, you have a guesstimate as to the time it should take. If most of your internal people give you an answer in 20 minutes it is unreasonable for someone under the much higher and more awkward pressure of an interview to consistently do that too.

Second, it's helpful to prime people some. Tell them in advance about the tool you will be using in your interview signup. Tell candidates what language they can/cannot use (I can program much better in Python than say Java, so I can solve problems in 10% of the time in Python than Java). Make sure they know what to expect. The tooling can be very awkward initially and you don't really want to reject people because of this.

And last, as you talk through the problems, encourage candidates to talk outloud with their thoughts. This will do a variety of things:

• Identify what is unclear about the problem so it can be clarified
• Separate people who are nervous and make syntax errors that they know conceptually from people who are clueless
• Give you more information about their skills than just the code

Answer 5

Since I have interviewed a few dozen candidates by means of a coding task, I think I might have some insight:

To assess real world programming skill, the test should be conducted a real world environment. That means an IDE, not a whiteboard. That means access to google and a quiet work environment, not being constantly interrupted by questions or advice from an interviewer.

Passing the test should not require a perfect program, because real world programs are not usually perfect at the first attempt. I am satisfied to see good progress towards a working solution. If bugs remain, I verify that the candidate is able to recognize the bug upon prompting, and can describe a reasonable fix.

To assess the difficulty of the test, do it yourself, or give it to some coworkers (mirroring interview conditions as closely as practical). Also look at the candidates you ended up hiring in spite of a marginal coding test. Did the test predict their actual work performance?

That said, even if your test is perfect it is entirely normal to meet the occasional candidate who can not program their way out of a paper bag. I don't think I'll ever forget the candidate who professed to be a Java expert, but somehow failed to get anywhere in the coding exercise. Curiously for such a poor performance, their browser window remained pristine - but they had forgotten to clear the browser history, which revealed google searches like "how to declare a class in Java". He did not get the job, but I had to repeat myself 4 times before the hiring manager believed me that the candidate who had talked so confidently about his vast experience in fact had no clue at all.

If you get many bad candidates, it is also possible the initial screening is inadequate. A total lack of qualified applicants might also mean that your job listing is misleading or insufficiently advertised - or perhaps there simply aren't enough coders looking for a job in your area.

Answer 6

When I used to select candidates to be hired, I used to make some live coding tests. One thing I used to say was: "feel free to use Google if you need to search for an inspiration". What I was looking for, more than the ability to remember by heart every libraries of a determinated language, it was the skill of "focusing the problem, building an idea about how to solve it, look for a way to implement it" That's what I really cared to test. I don't really mind if they will look for a similar approach in Stack Overflow. With this approach, I didn't care so much about how much the problem could be more or less (always considering, of course, that the challenge should be accomplished in a reasonable amount of time)...

Answer 7

IMO, templates and skeletons to fill are more confusing than simple task from scratch, especially when we talking about 3-4 lines solution. How about step-by-step algorithm description task? Everyone in the industry can do it, if he can call himself programmer. And if candidate can write algorithm, he can implement it.

Perhaps they think you try to coax the usable work out of them under pretense of test task.

If you keep task short and simple with no seeming connection to actual usability, you shouldn`t have issues.