42

In engineering fields it can be quite difficult to come up with good, consistent questions that enable an interviewer to consistently get a good overview of what a candidate is capable of. Unfortunately, in these fields, recruiters are very desperate to get placements because it can be quite lucrative, and so they do whatever they can to give their candidates an edge - including quizzing them on the questions that they were asked so that they can prime their candidates for success at answering the questions. The end result is that our questions become useless, as if a candidate already knows the answer (or has a chance to research it ahead of time) we can't get any data about their actual ability to solve problems themselves.

Short of having to come up with new questions for every candidate (bad because it increases the interviewing workload and makes it difficult to make objective comparisons) or asking HR to bar candidates who come from these bad recruiters (they are unwilling to do that), how can we avoid having recruiters "poisoning the well" in this way?

EDITED TO ADD: The questions we ask are not simple "memorization" or "trivia" questions. They are problem-solving and high-level systems design. However, this question still applies to fields where memorization/trivia questions are an important part of the screening process, and the nature of the problem itself is not the part of this situation that needs to be changed in any case.

Also, the recruiting firms are hired by us to find candidates - we're the ones paying their hiring bonus, not the candidates. Thus, they really should not be sending us low-quality candidates that they have prepped to seem like high-quality candidates.

  • 27
    An interview should not just be questions. It should be a conversation, using some questions to start that conversation. – Ed Heal Oct 27 '15 at 19:07
  • 4
    @EdHeal The OP is completely valid/on-target. If you want to go create a pool of dozens of distinct questions that is admirable. But unlikely. So for someone who does not have infinite time on their hands -and who justifiably places heavy emphasis on coding ability - this is real conundrum. A comparison would be if you were the TA for a class and needed to generate many sets of quizzes because the answers keep getting leaked. Not optimal/fair use of your time and effort. In addition it does detract from apples to apples comparisons. – javadba Oct 28 '15 at 0:11
  • 2
    @EdHeal You're assuming a lot about our interview process. – fluffy Oct 28 '15 at 1:25
  • 5
    I think the problem is with the recruiters your company is using. – DA. Oct 28 '15 at 4:46
  • 2
    @Brandin Often the recruiters will speak to the applicants afterwards (at least that is what i encountered being an appcilant brought together with a company through a recruiter) and ask them what where the topics of the job interview, what questions where asked and if there was anything said about wages or similar. – Aron_dc Oct 28 '15 at 9:32
46

What questions are you asking?

I'm assuming you aren't asking trivia questions but have more meaningful questions.

If you have design types of questions, you can switch small details and really trip someone who doesn't actually know what they are doing up. Changing small details of your interview questions can really prove a few things:

  • Non-qualified candidates who "studied for the test" will fail miserably
  • Qualified candidates who know the material will easily be able to adapt and thus prove themselves

If you don't want to do this, ask lots of "why" questions. "Why are you doing X?" "What about Y?" as followups. Good candidates will be able to answer, or at least discuss intelligently, regardless of preparation - bad candidates won't.

... have you talked to the recruiter?

You may also want to ask your recruiter about this and see why they are doing it. That may be not aware it is causing you problems. You may quickly resolve this by a short conversation.

Otherwise, if they tell you they aren't/refuse/whatever, get better/different recruiters. It sounds like you are working with bottom of the barrel recruiters.

Also consider the possibility your questions are posted online (and not from the recruiter). I'd suggest first looking into this before blaming your recruiter. If you work for an even remotely high profile company chances are questions are posted all over the Internet.

You can also consider including NDA type of agreements in your interview process. If you have a dedicated HR/legal department approach them about this issue.

  • 13
    I disagree with the "bottom of the barrel" remark. A recruiters' job is not to make sure the best candidate gets your job, it is to make sure their candidates get the job. That's their job. Just as a lawyer's job isn't to see justice done, it's to give their clients the best defense possible. – Francine DeGrood Taylor Oct 27 '15 at 18:06
  • 10
    @FrancineDeGroodTaylor any recruiter who blatantly copies questions and gives them to candidates without asking/telling the company is... not really doing a great job of providing a quality service to fluffy's company. – enderland Oct 27 '15 at 18:10
  • 8
    There could be any number of reasons for why they provide questions to their candidates but I don't see how that particular discussion is relevant to this question and Enderland's excellent answer. My take on it is that if you have good and useful questions and a good interviewer it won't matter that the questions are publicly available. – Lilienthal Oct 27 '15 at 18:25
  • 2
    @FrancineDeGroodTaylor - I strongly disagree. A bad recruiter is there to collect their commission as fast as possible. A good recruiter provides a service. The fact that people even consider the former, however, is a sign of how truly woeful the recruitment industry is. Having encountered both good and bad recruiters, the difference is stark. – Jon Story Oct 29 '15 at 16:11
19

Instead of coming up with different questions for every candidate, come up with questions that are going to be different for every candidate. The best interviews I ever had involved an open ended project "Show me how you would design a game of monopoly" "write a method to check to see if a string is a palindrome" "I'm a client who wants you to design a system for them...ask me questions and figure out what I want", etc.

You don't have to do a lot of footwork to simply swap between monopoly and solitaire (or whatever the engineering equivalent would be), and what kind of prep work are they going to do when the test is basically "show me how you go about doing engineering work"?

An example might be to come up with a list of issues that you've run into in the job that you will be asking him to do. Ask him how he would go about troubleshooting and solving the problem. Listen to his answer and judge him on it, but don't tell him what the "real" answer was. So what if the candidate might have a little extra time to research the problem...that's more realistic than expecting him to know it on the spot. Real engineers solving problems have the same time and resources that your candidates will have if they have advanced warning of the nature of the problems.

And if you really want to level the playing field, pass out a list of real world problems to the candidates a day or two beforehand. After all, what you really should be testing is their ability to use the resources available to solve problems, not their ability to have facts memorized.

  • The problem with switching out different scenarios in the same general theme is you won't be able to reuse experience about how hard the question is, where the common misunderstandings are and how to avoid them, where the hidden pitfalls are (and whether the interview is best served by steering towards or away from them). You can survive without that stuff, but it makes your job much harder than it needs to be. – Ben Millwood Oct 28 '15 at 14:54
15

Start asking questions the recruiters can't answer. Otherwise, the questions are just like Trivial Pursuit or Let's See If You Know What I Know sort of games.

Questions based on experience or require the ability to apply knowledge are going to be difficult for recruiters to help with too much prep. There's nothing wrong with someone preparing themselves anyway.

You have to decide the importance of memorization in your field. Maybe you should have the candidates do a small task that is typical of your projects to demonstrate they can do what the job requires.

  • 20
    +1 I really dislike being asked canned questions that have a researchable answer. It's objective, but it measures the wrong things. Ask questions that don't have a specifically correct answer. Ask candidates to describe a problem they ran into and how they solved it. Questions like that can be prepared for, but can't easily be cheated on if the folks interviewing know the area they're asking about and ask follow up questions based on what has been said. – ColleenV Oct 27 '15 at 17:32
  • 2
    This is also a much better approximation of the work they will do for you. After all, when was the last time you or one of your developers actually coded an insertion sort? :) – cdkMoose Oct 27 '15 at 17:34
  • @ColleenV, you're absolutely right. That's not an interview, that's Trivial Pursuit. Hate those. – Nolo Problemo Oct 27 '15 at 19:08
  • 3
    We are most definitely not asking memorization-type or trivia questions. We are asking high-level problem-solving questions that one isn't likely to have come across before. And of course we try to adapt the question based on how the candidate is doing. But if someone has a week to prepare for every aspect of a question and memorize existing solutions to it, that highly dilutes our ability to differentiate good candidates from bad. – fluffy Oct 28 '15 at 1:28
  • 2
    I also hate interviews that put you on the spot like that. Problem solving or technical abilities are better assessed through testing or some other means in addition to interview. An interview is just a chat to get to know the candidate personally and see how they behave/work. – Luke Oct 28 '15 at 12:13
5

Make sure your contract with the recruiters states that the bonus is only paid if the person stays in your company for, say, 3 months. If a candidate comes through your screening process by memorising questions he/she will probably not be with you for long...

  • This has been part of the contract with recruiters everywhere I've ever worked (sometimes on a sliding scale). UK based, for what it's worth. – Philip Kendall Oct 28 '15 at 19:45
2

The other answers give short shrift to the severity of the issue presented by the OP. The OP is valid/on-target. If you want to go create a pool of dozens of distinct questions that is admirable. For an interviewer who does not have infinite time on their hands -and who justifiably places heavy emphasis on coding ability - this is real conundrum.

Given all this - if I were the interviewer I would put a significantly higher bar on those candidates coming from that particular agency. The actual questions would likely be more off the cuff: in any case they are going to be "leaked" back to their agency so they do not deserve as much attention.

You can probably discern in any case if the candidates had heard the questions before.

  • 3
    The other answers are fine. None of them suggest creating pools of dozens of answers; they all suggest either modifying details in the questions or using more open-ended questions that require more than a memorised answer. Trying to come up with an off-the-cuff question for a candidate you suspect may have been fed the answer to a previous question, is much more difficult to do consistently than preparing a good question ahead of time. – Logan Pickup Oct 28 '15 at 1:23
0

Rather than asking them to come up with a design (which is the kind of thing you can learn for), let them look into someone else's existing design and perform a peer review of it. That's a very different skill, a very important one and you can just pull up the last few commits made to Github on some public project and review that with the candidate. There's literally no way to prepare for a code review of a piece of code that didn't exist yet when you started the interview.

(This of course requires that the interviewer be at least at capable as the interviewee, because he'll also have to be able to analyse an unknown codebase in fairly short time.)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.