27

I am a senior software engineer and a lead developer in a company with 250 employees. The company is hiring mid to senior level developers. I am doing technical interviews, even though I have been here for only 2 months. I don't have much prior experience doing interviews as my previous company was much smaller and hiring new backend engineers was very rare.

A part of the interviews is live coding, usually through Skype. The candidates can use their own IDE or use a collaborative real-time editor, such as CollabEdit. They have 20 to 30 minutes to complete the task. I do not care about the syntax, the point of the task is to prove the ability to find the solution and implement it.

A lot of promising candidates (having good experience, knowledge and ability to evaluate why a particular framework/technology is used) fail on what I think are easy coding tasks (for example, finding the second biggest number in an array). I understand that they are under pressure, but I try to guide them and make them feel comfortable. I usually also provide them with code skeletons to get them started.

I have consulted with the Technical Lead. He says if the candidate is unable to do this, we should not continue, as we are looking for programmers and we expect them to write code.

Since so many promising candidates are failing these coding tasks, I am wondering if they are really as "easy" as I had assumed. How can I determine if a coding task asked in the interview is too hard?

  • 1
    Comments are not for extended discussion; this conversation has been moved to chat. – Monica Cellio Nov 8 '17 at 16:49
  • Design the a task that can be done by any developer but will gives a lot of opportunity for a good developer to do a better job. – Keith Loughnane Nov 10 '17 at 14:53
  • 1
    Mid to senior level developers - if they can't sort the array of numbers large to small and get index 1, then they are obviously not qualified for the position... – Chris Cirefice Nov 11 '17 at 9:20
  • You don't even need to sort. You just parse the array once and keep the largest and second largest number. I consider myself to be a senior developer, and this took me 6 minutes to implement in a language I'm slightly unfamiliar with: had to search for an example of array parsing, get an initial working solution, realise it has a bug, and fix the bug. During my PhD I used to TA "data structures and algorithms". We had undergraduate students implement harder problems, on paper, for a fraction of the grade. Granted, some of them didn't do the coding part, but I wouldn't advise OP to hire them. – V N Jan 30 at 15:07
29

How can I determine if a coding task is too hard for an interview candidate?

Your question is ambiguous. Let me try to crisp it up. I think what you are really asking is:

We have a process for obtaining a signal from candidates about their fitness. If the process produces a "no hire" signal for candidates that are actually fit, then the technical question portion of the process might be too difficult. How can I determine if a question is too difficult?

That's easy; as others have said, put your coworkers who are known to be fit through the process; if the process consistently rejects them, then there's something wrong with the process.

But that's not the problem you have. This is a question-and-answer site, and you've asked the wrong question.

The problem you actually have is a significant fraction of candidates are plainly unfit, and this is wasting time, effort and money that could be better spent elsewhere.

You have a pipeline problem, not a coding question problem. Something in your candidate pipeline is producing a large fraction of plainly unfit candidates. As is well known, a great many programming job applicants have no ability whatsoever to write even simple programs: https://blog.codinghorror.com/why-cant-programmers-program/

Ask around and see if other interviewers are having similar problems. If they are, then push back on management and recruiting to see if they can figure out what is incentivizing recruiting to keep sending you unqualified candidates.

  • 1
    Your solution assumes strongly that the coding task is an appropriate difficulty for this interview, which is the one thing you can't assume. If the coding task is indeed too difficult then we can assume nothing about the fitness of the candidates. – TheSoundDefense Nov 7 '17 at 22:11
  • 10
    @TheSoundDefense The example coding task is is to find the second largest number in an array... dead simple to anyone who could legitimately claim to be a professional programmer. – Maybe_Factor Nov 7 '17 at 23:23
  • 2
    @TheSoundDefense : I'd go even further than MaybeFactor, and say that it is damn easy. Not only the candidate should be able to make it, but under heavy pressure as well. Said otherwise, finding good candidates is hard, and sometimes, it's better not to hire anyone than to hire an unfit person because "we have to take someone". – gazzz0x2z Nov 8 '17 at 15:21
  • 11
    @gazzz0x2z: I disagree with your "sometimes". It is ALWAYS better to not hire someone than to have an unfit person. Having no one means their work doesn't get done. Having an unfit person slows down everyone else in the organization too. You never have to take someone. – Eric Lippert Nov 8 '17 at 15:31
  • @EricLippert : you're right, I've been too cautious in my comment – gazzz0x2z Nov 8 '17 at 15:53
12

How can I determine if a coding task asked in the interview is too hard?

Seems that those failing candidates are evidence to support that your coding task may more challenging than you think.

Empirically, you can determine if future coding tasks are "too hard" if you see several candidates failing or taking too much time doing them.

You can also have yourself or some technical coworkers to try do the coding tasks, take your time doing it, and then compare your results with whatever your expectations of that particular tasks were. You can then determine if it needs some simplification to be able to complete in a regular coding interview.

Keep in mind that, usually, coding interviews or tests should not be too long. Otherwise you will not be actually evaluating the knowledge on the matter, but instead just seeing who is better in pretending to know a programming language (in 2 hours anyone can search and become an "expert" on any Stack Overflow tag). Usually, simple and straightforward questions or tasks are more effective in measuring the language knowledge as well as some creativity and improvisation.

I also suggest you take a look at this question, where several answer are given that could help decide if a specific task should be removed or not.

  • 19
    +1 for asking co-workers to do this. If a co-worker fails, OP also needs to ask "would I fire this person?" If the answer is no, the task either is too difficult, not appropriate to the position, or should not be the deciding factor. – Kathy Nov 7 '17 at 17:21
  • 1
    Agreed, @Kathy. If all viable candidates should be able to answer your interview questions, then it stands to reason that everyone who is actually employed there should also be able to answer them. If they can't, then you should probably re-calibrate. Plus, asking coworkers helps ensure that your questions don't make it out to the rest of the world, which would make them less useful to you. – TheSoundDefense Nov 7 '17 at 17:24
  • @Kathy that's right. When defining coding tests you have to try them first before actually implementing them. Thinking and imagining a coding task is one thing and actually doing it is another and that degree of difficulty should be considered at all times. – DarkCygnus Nov 7 '17 at 17:25
  • 1
    I would consider a large fraction of candidates who fail to complete trivial tasks in an interview setting to be evidence that there is something deeply wrong with lead generation and initial screening by recruiting, not evidence that the process is too hard for qualified candidates. – Eric Lippert Nov 7 '17 at 18:31
  • 2
    @EricLippert also true, that or that there is a misunderstanding in what Screening looks for in candidates and what the code interviewers are asking them to solve, it could be that those objectives are not aligned, but that would be speculating without further feedback from OP (in which case could probably belong to another question) – DarkCygnus Nov 7 '17 at 18:35
9

I had similar problems, with very simple coding questions. It seems that many people are what we termed “wizard's apprentices” after the IDE code-generator wizard and the Micky Mouse cartoon. They think they know C++ but really have no idea how to write a class, even a simple one.

Here is the first one. I always read it so as to give the exact same wording to everyone, but from memory now: “Write a class whose constructor takes two integral arguments and has members to return their sum and difference.”

I expected to see different levels of how well they scrutinized the specs to nail things down before starting, and people who used templates rather than choosing a specific type, etc.

What I got was a surprising number of people who could not write a class, period.


for example, finding the second biggest number in an array

As I said, nail things down before coding:

  • Can I modify the array?
  • Did you mean the second element in sorted order, or do you mean the second largest unique value ([1,2,2,3,3,3]is the answer 3 or 2?)?
  • Should it deal with arrays that have fewer than 2 elements (or unique elements, depending on answer to prev)? Throw exception, plan for optional return value, or what?

Should I clarify the specifications (and get signed off on it) and then write critical test cases, as in "real life"?

Your stated goals are at odds with the unclear specification. As stated, you really should test engineering skills first, then coding skills. And do you really want to hire someone who will throw together an unreadable overly-clever un-maintainable mess? I question the wisdom of not caring about syntax.

If you just want to know if they can approach the problem, do orals rather than a coding task. Have him explain how he’d do it: e.g. I know about the standard algorithm, so duh. If that does what you really wanted. Be impressed that I have useful references bookmarked and always open, and don’t worry about the final step of writing a sample program that contains a line that calls this function.

So I question whether your test is really ready for someone to just code, or if the issues I pointed out would paralyze the candidate who is told that this is a complete specification ready to code?

  • As an interviewer, I use unclear specs to see how the candidate resolves ambiguity. Do they ask clarifying questions? Are their assumptions reasonable? Resolving ambiguous specs is something every experienced coder should have experience with, so testing that skill seems fair. – meriton Nov 8 '17 at 5:11
  • @meriton yes, I agree completely. That's why I’m telling the OP that asking for hacked-up working code (only) from unclear specs may be part of his problem. – JDługosz Nov 8 '17 at 18:22
7

How can I determine if a coding task asked in the interview is too hard?

There are a variety of ways. By far the easiest is to find a community of people you want to hire and ask them.

This might be a meetup type group. This might be internal people. Maybe online forums. For programmers you will get tons of feedback because most programmers can easily be nerd sniped with "hey, can you help me see how long this programming question takes?"

Figure out standard introduction to your questions, too, so everyone gets the same information. Maybe a talk sheet for the problem intro. Interviewing can be stressful for both parties.

Then, once you have a time for the task, multiply it by at least 1.5 to account for nervousness inherent in interviewing and bam, you have a guesstimate as to the time it should take. If most of your internal people give you an answer in 20 minutes it is unreasonable for someone under the much higher and more awkward pressure of an interview to consistently do that too.

Second, it's helpful to prime people some. Tell them in advance about the tool you will be using in your interview signup. Tell candidates what language they can/cannot use (I can program much better in Python than say Java, so I can solve problems in 10% of the time in Python than Java). Make sure they know what to expect. The tooling can be very awkward initially and you don't really want to reject people because of this.

And last, as you talk through the problems, encourage candidates to talk outloud with their thoughts. This will do a variety of things:

  • Identify what is unclear about the problem so it can be clarified
  • Separate people who are nervous and make syntax errors that they know conceptually from people who are clueless
  • Give you more information about their skills than just the code
5

Since I have interviewed a few dozen candidates by means of a coding task, I think I might have some insight:

To assess real world programming skill, the test should be conducted a real world environment. That means an IDE, not a whiteboard. That means access to google and a quiet work environment, not being constantly interrupted by questions or advice from an interviewer.

Passing the test should not require a perfect program, because real world programs are not usually perfect at the first attempt. I am satisfied to see good progress towards a working solution. If bugs remain, I verify that the candidate is able to recognize the bug upon prompting, and can describe a reasonable fix.

To assess the difficulty of the test, do it yourself, or give it to some coworkers (mirroring interview conditions as closely as practical). Also look at the candidates you ended up hiring in spite of a marginal coding test. Did the test predict their actual work performance?

That said, even if your test is perfect it is entirely normal to meet the occasional candidate who can not program their way out of a paper bag. I don't think I'll ever forget the candidate who professed to be a Java expert, but somehow failed to get anywhere in the coding exercise. Curiously for such a poor performance, their browser window remained pristine - but they had forgotten to clear the browser history, which revealed google searches like "how to declare a class in Java". He did not get the job, but I had to repeat myself 4 times before the hiring manager believed me that the candidate who had talked so confidently about his vast experience in fact had no clue at all.

If you get many bad candidates, it is also possible the initial screening is inadequate. A total lack of qualified applicants might also mean that your job listing is misleading or insufficiently advertised - or perhaps there simply aren't enough coders looking for a job in your area.

2

When I used to select candidates to be hired, I used to make some live coding tests. One thing I used to say was: "feel free to use Google if you need to search for an inspiration". What I was looking for, more than the ability to remember by heart every libraries of a determinated language, it was the skill of "focusing the problem, building an idea about how to solve it, look for a way to implement it" That's what I really cared to test. I don't really mind if they will look for a similar approach in Stack Overflow. With this approach, I didn't care so much about how much the problem could be more or less (always considering, of course, that the challenge should be accomplished in a reasonable amount of time)...

  • 2
    although you have a point, I think this does not answer the actual question made here – DarkCygnus Nov 7 '17 at 18:42
1

IMO, templates and skeletons to fill are more confusing than simple task from scratch, especially when we talking about 3-4 lines solution. How about step-by-step algorithm description task? Everyone in the industry can do it, if he can call himself programmer. And if candidate can write algorithm, he can implement it.

Perhaps they think you try to coax the usable work out of them under pretense of test task.

If you keep task short and simple with no seeming connection to actual usability, you shouldn`t have issues.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.