15

I am curious what people's thoughts are on various types of testing in the interview process.

For the sake of argument, let's start with the following as given:

  1. Interviews are a very sub-par way to determine good employees from bad. (It can be hard to tell if someone is worth hiring after a 10 week internship. After a 1 hour interview it is nearly impossible)

  2. Interviews are expensive if you haven't pre-screened well.

  3. There is a lot of bias in the interview process.

  4. With grade inflation, academic credentials aren't what they used to be.

  5. If you don't know the person doing a reference, it likely isn't useful.

So my big question is - what types of testing can be worthwhile (and legal) in this context?

More specifically: A lot of people do whiteboard coding for programming interviews, but that doesn't solve #2 and #3. Can a written exam given to a larger subset of people help? Are there HR/legal issues to be worked around? Also - are there other "subset of a job" techniques that people have used in interviewing? (Make a presentation, show me something you've written, etc.)

For the sake of guiding the answers, can you comment on thoughts specific to written exams or other non-standard interviewing techniques? Thanks!

  • 4
    Of course interviews are expensive if you haven't pre-screened well! So... pre-screen well. – Angelo Jul 2 '12 at 13:31
  • 2
    I do not know if i agree with If you don't know the person doing a reference, it likely isn't credible. While the reference may not translate to will work well for you that does not mean the the reference was false. – IDrinkandIKnowThings Jul 2 '12 at 14:21
  • @Chad - Fixed. Your point captures my initial intention better than how I wrote it. – MathAttack Jul 2 '12 at 22:41
  • 1
    I think all your assumptions are true, except #1. There is nothing better than a face-to-face interview. – kevin cline Jul 10 '12 at 15:49
  • 1
    IMHO whiteboard testing is strictly inferior to sitting someone down with a laptop and the exact conditions/framework that they will be working with and asking them to solve a real problem. For example, if it's a book company ask them to code an ISBN number-related task rather than FizzBuzz. – MrFox Oct 15 '12 at 15:47
13
+50

Interviewing is more art than science. Some people are not good test takers but thrive under real life pressure. Others are great test takers but fold in real world conditions.

The most important thing is to find a method that works for your group.

The most important thing is to get the right number of candidates to each phase screening process. If you get too few to any one phase then you start feeling pressure to lower standards and expectations to fill positions. This often leads to poorly qualified people being over their heads and unable to perform. The result is a negative for both sides. If you get to many then it is difficult to make a good choice and you end up turning away great candidates many times unrealized. What is the right number is a subjective matter that varies. You need to find the right numbers for your organization

Most commonly I have seen a 4 phase process.

Prescreen: An initial check to make sure the applicant meets the bare minimum requirements. There is no point in interviewing someone that can not be brought on anyway. Some companies have used an automated test here as well.

Phone Screen: The initial interaction with the company, simple tech test, introduction to the company, verification of expectations and next steps.

Skill Test: Gauge the skill level of the candidate. I have seen the range of FizzBuzz to a Kobayashi Maru type(there was no way to actually solve it just about how you handle the test). Personally I dislike the unbeatable test because I think it sets a bad tone for the relationship. I prefer that candidates come on board with a positive experience of success rather than failure but that is my preference.

Team Interview This is a test of the culture fit. This is not so much about what they know but how they interact with the key people on the team. Technical questions are asked not to probe knowledge so much as to gauge levels of interest and compatibility.

Hiring is expensive. Made more so if you hire to many people who do not work out. There is no golden bullet but getting the right number of people to each phase is important to keep your team/business healthy.

  • 1
    @MathAttack - As I said im my answer I do not think you can adequately evaluate a candidate with just a test. I think a test is an important part of the process but if you rely on the results of tests then you end up with a company full of good test takers... that does not always translate to good workers, and certianly not to "the best possible staff". – IDrinkandIKnowThings Jul 6 '12 at 12:26
  • 1
    @MathAttack - No it is an important tool. Just not a silver bullet. There is a reason that the 4 phase process is popular... it is more successful than many alternatives. That said each company has its own implementation. It is an imperfect process because people are imperfect. Define your process and tweak it till your find a mix that works for you. – IDrinkandIKnowThings Jul 6 '12 at 14:01
  • 1
    @Chad: Could you please explain what you mean with you disliking FizzBuzz but at the same time prefer that they come on board with a positive experience of success. The Kobayashi Maru test is a test of character in an experience of failure (no-win scenario). You might need to edit that. ;-) – Spoike Jul 8 '12 at 8:57
  • 1
    @Chad: Yeah, you wrote "former" but meant "latter" which made it confusing. Sorry to nitpick, edit ok! – Spoike Jul 8 '12 at 13:08
  • 1
    @MathAttack, I don't think you can ever truly expect to evaluate candidates totally "objectively". Evaluation of fit and potential for job performance requires a strong component of subjectivity (and there's nothing wrong with that). However, I think if you do come up with a test written or otherwise, it would be a good idea to apply the test on current employees to see how your top-performers score and compare that to the candidates. – Angelo Jul 8 '12 at 18:15
10

Pre-Evaluation: Ask Them to Rate their Skills

I ask applicants to rate themselves for all the technologies on their resumes according to this scale:

 1 - Read the book on X.
 2 - Reviewed the book on X.
 3 - Wrote *THE* book on X.

Make it very clear that this is to be taken very close to litteraly.

Inevitably, a majority of people will send back evaluations with at leat one 3 and a few 2, but not a majority one 1s.

Of course, you can't determine and evaluate someone's skills based on these raw numbers and their own evaluation. But it establishes a trust contract, and many are keen to break it. Or incapable to read a simple request and stick to it. When you do your phone screening, keep this evaluation handy, and if after 2 questions they are obviously one level down in most of their key technologies, they were overestimating themselves. And if they're below the level of acceptance you've set for the role, just call it a day. There's no guarantee they could be still good people, but you have better things to do.

It's a nice approach, as it forces people to really reflect about their skills. The good ones are the ones who would have put their level as "advanced" on their resume, and suddenly realize what "advanced" really should mean, and do have humility to backtrack.

The issue with this approach is agencies. They are usually annoyed by this, as they make their money from finding people, and you just lengthen their process, which may already be long, and basically might scare off some people.

Pre-Test: Be Challenging and Strict

I don't mind giving pre-screening exercises. It's controversial as not everybody will have the time to conduct them, and some can cheat on them, but you just had some barriers to resolution. For programming roles, it's especially easy to do as you can request applicants to submit programs that can be marked by a program or evaluated quickly. You'd need to write these programs once, and then you're done. It's a good time investment.

Then send out these exercises, and make sure the guidelines are crystal clear on what output you expect. Rigor is something of great value with developers, so you want to find the ones who CAN READ A BASIC ASSIGNMENT AND IMPLEMENT IT WITHOUT WHINING. Writing small programs that return some numbers or strings and comply to the assignment exactly (no EXTRA characters here and there) is easy.

Just with that, I often weed out non-motivated applicants, or some take offense. It's fine. It's communication. They're basically telling you, knowingly or not, that they weren't that interested. You just saved time on the next interviews.

For the ones who send something back, if some send exactly what you want, be sure to keep a close eye on these. They aren't that common, though they should be. For the ones who fail entirely, you can probably dismiss them. For the ones who failed to follow the guidelines by a small margin, still get them through, if you can: it's just one more rating point you can use in your decision taking process later, even if it's not entirely conclusive at the moment.

These need to be simple (in terms of implementation and time it takes) assignments, but with a problem-solving touch.

Phone Screening: Be More Flexible, Cover A Lot of Ground

If this is for a short phone screen, put them at ease (the usual yadda-yadda: "drive us through your resume with emphasis on what you think matters for this role", "what are you looking for", and describe a bit more about the role and ask if they have questions.

Don't make it last too long, as otherwise they're getting nervously tired, so get it out of the way quick, and start the phone tests.

Ask clear, short questions, and make sure to be more flexible than for pre-tests. They are nervous, so give them time, and lead them towards the solution. You want to detect a "mindset" here, and a "personality", not raw skills.

If you have time for a long-ish phone screen, use online editors with collaborative capabilities (EtherPad, Google Docs...), and help them through problems. Keep backups of these for the later stages to refresh your memory about applicants and be able to review all of them once you've seen a few: you might revise your judgment based on the individuals you've rated so far. Some EtherPad clones even allow to record the session, so you can do a play by play of the session's typing, which can be very interesting. When you ask a question verbally, make sure to have it already pre-written in another editor to copy-paste it (and possible code stubs, comments and examples) in the shared session to not waste time.

On-Site Test: The Employee-Experience

For face-to-face interview(s), you want the whole shebang:

  • resume review,
  • whiteboard brainstorms,
  • technology-centric discussions,
  • informal chats (around water-coolers or even over lunch),
  • hands-on tests.

All of these are rather self-explanatory, but the last one has some specifics: do have a computer ready with pre-installed tools, and assignments. Make sure to have the capability to later record the applicant's work. I'm not saying to live record, but just sending the coding assignment to a version control system or document management system is great for future reviews, especially if candidates re-apply later.

Involve multiple aspects of the daily job in the coding test, and have some paper-based and computer-based components:

  • paper-based code-review,
  • live code-review (by rewriting the code, or annotating it in a Crucible or ReviewBoard if you have one with a guest account),
  • live coding-test (make sure that this includes writing code AND writing tests for the code. Or writing test for already existing code. Or writing code for existing tests.)

Debrief quickly with them so they can run you through it, ask for final closing statements and questions, and call it a day.

Make sure that multiple team members take part to the tests and interviews, possibly some senior developers and some junior developers or even interns (great training for them), and even people from other departments (QA, training, support ...), if only for a few minutes. You'll all end up working together. So while it's important you to have the final decision, it's good for other to be able to weigh. Make sure you rotate these panels of applicants during the different phases of the interview, with some elements staying throughout: this way you can avoid possible conflicts with the odd employee who really can't stay a guy (though, in the end, if that were the case, you unfortunately probably should pass even on a good applicant).

Obviously, I rarely interview more than 3 applicants per day. More likely 2: one in the morning, one in the afternoon, or make them overlap in the way that's not annoying.


How to Keep it Cheap

All this seems considerably time-consuming. And it is, and you want to invest some time in finding long-term hires, for a great ROI. But, if you automate most of this work, and make it part of your process, it takes away a lot of the routine and guesswork, and you can get pretty efficient at it. Of course, at some point it even gets a bit robotic, which isn't too good, but it's hard to avoid.

The pre-evaluations and pre-tests seem harsh and like they could weed out or scare away very good potential candidates, and that's probably the case. Maybe some of these who give up would have been great hires if they had gone through. But the cost of dealing with all the other ones who wouldn't have been great hires is far more threatening and depressing thatn the perspective of not seeing a stellar employee walk through the door. And the process aims at finding you other stellar employees. So don't think of those giving-ups as lost potential hires, but as time-savers, giving up space for the next big ones.

  • 2
    "They're basically telling you, knowingly or not, that they weren't that interested." Is your company as in-demand as Google, Microsoft or Thoughtworks? Cause if not then why should they be that interested, at the beginning of the process, when many other jobs are also available to them? Convince me that I want this job more than the others before you start asking me to spend significant amounts of my time convincing you that you want me. – pdr Jul 5 '12 at 7:45
  • 2
    @pdr: Do you think everybody wants to work for the big "stars"? Not really. Many people are interested in specific fields, in which case they are interested in actors of this field. Not everybody wants to be naked and famous. If your company doesn't look interesting on the outside, that's another problem: your issue is not in the interview process, but in selling the job offer, advertising your company's work culture, and marketing in general. That's where it starts. NOT during the interview process. Plus, as I told, you SHOULD NOT think of it as "lost" potential employees. Others will come. – haylem Jul 5 '12 at 7:50
  • 3
    There are not enough genuinely good developers out there to dismiss them that way. Yes, a false negative is better than a false positive, every time, but "Others will come," dismisses a very large opportunity cost. And no, not everyone wants to work for the big stars -- I don't -- but they have enough people fighting for places that they can afford to put them through a gruelling interview. Most of us don't. – pdr Jul 5 '12 at 9:44
  • 1
    @pdr: The company I currently work for might be big enough to not care about this cost, according to your theory. But the startups I worked for definitely weren't. They were probably just attractive enough that it made this irrelevant (like I said, it's upfront marketing of your culture that counts), or maybe we were just very efficient in dealing with this. Maybe you just need to refine your process to make it more lightweight and less costly then. – haylem Jul 5 '12 at 10:54
  • 2
    @Spoike: Have you read tutorials, online blog posts or lurked around P.SE? Well that's good enough. It's not necessarily exactly the same thing that you need to look for, otherwise it's just as bad as dismissing someone for not knowing algorithm X or Y. That's why I said it's too be taken close to litterally. It's not necessarily exactly these actions that you need to have done, but equivalent ones. Obviously it's a simple scale, but you know what to answer. I'm sure you did, you just wanted to nitpick (which is not so well received for interviews. Arguing is fine, nitpicking is not ;) ) – haylem Jul 6 '12 at 17:43
4

For tests on written work I would focus on:

  • Their code samples. Go through the examples they provide with them and talk about the approach and the details.

  • Their network. See who their peers (via linkedIn, Stack Overflow, etc. are) and what kinda reputation they have for code.

  • Whiteboard development of some simple code. I generally prefer to stay away from much hands-on machine work as people can be nervous on the keyboard and/or unfamiliar with tools you want them to use. Smart folks will learn the whatever the setup is, so focus on their approach for screening.

  • 4
    What if the candidate does not have a network like this? Are you suggesting that if someone does not play in the social network sandbox they can not be a good employee? – IDrinkandIKnowThings Jul 9 '12 at 20:18
3

From what I heard a very good technique of testing potential programmer employees is to schedule some time for video conference and ask them to solve some simple problems in a specific amount of time. This approach doesn't take too much time (and can be streamlined with some automatic tools if there are a lot of interviews), and allows the interviewer to screen people that don't have the right skills before the proper interview.

As far as I remember the technique was used by some respectable company and the results were quite astonishing - a lot of programmers applying for their positions failed such basic tests. This allowed them to weed out a lot of candidates and leave room for more proper interviews for the ones that passed this "bare essentials" test.

3

Are you considering behavioral interviewing in your process? Research appears to support its efficacy over theoretical test questions, especially for high-level jobs; see for example Motowidlo et al., Pulakos and Schmitt (summarized by Huffcutt et al.). The former article correlates information from recorded reports (not just live interviews) with judgment validity. I believe this would apply to written behavioral interview tests, although I have only practiced behavior interviewing in person. There are probably established guidelines for a written approach if you choose to take it. If you go with more of an in-person route, the research suggests interviewer skill is an important factor.

  • I'm considering behavioral interviewing as part of interviewing. It's better than regular interviewing, but still has bias. Similar to case interviews for strategy consulting firms - if someone likes the interviewee they can pull them through it. – MathAttack Jul 4 '12 at 12:26
3

A way test people and find out if people are going to fit with your team is to involve the candidate in the real process that you have with your team instead of inventing scenarios just for the interview or coming up with trick questions that supposedly tests the knowledge that they have. This makes for a better team interview, because unless your team interviews people often it turns out they are probably ill-prepared and very bad at interviewing people! Afterall, do you sit around with your team on a regular basis, singling out one person to ask them random technical trivia questions? That's the way most team interviews are conducted and it feels alien and isolating and nothing like the actual working environment. So, make it real...

During the later stages of the interview process treat them as if you hired them as a consultant to work on a specific problem for part of a day. At the later stages of the interview process both you and they should be willing to commit a good 2- to 4-hour (or more!) chunk of time to the interview process. That should be sufficient time to involve them in a real meeting with the team members they would be working with to discuss and begin working on an actual problem your team is trying to solve. Yes, that takes quite a bit more preparation and planning on your side, but it will tell you a lot more about how someone thinks and works along side others than any number of interview questions you might ask.

The hard part of this is carving out a well-defined problem that is solvable, at least in part, in a reasonable amount of time, that is deep enough to touch on multiple technical areas, and that requires enough interaction with multiple members of your team to get a sense of how they approach problems and work with others to solve them.

Working along side others is much better than isolating them in a room with a problem to solve since their communication skills and ability to work with your team is often more valuable than any domain specific knowledge they may bring with them; not that the latter isn't important, but it's not worth hiring the smartest person in the world if they're gonna act like a jerk and can't work with everyone else.

Something that makes this process easier is if you already use pairing stations with your current team, as is popular these days in some work places. Note I said pairing station rather than pair programming station, because a pair doesn't have to be limited to just writing code-- other office disciplines can benefit from that kind of setup, as well. I don't recommend using a pairing station just for the interview, though! If you don't already use them in your workplace with your team, then it will be another artificial part of the interview. For the team that I manage we don't use pairing stations for everything, but it is part of our office setup and I make sure that we make use of them when it is effective to use them, so in our environment it makes sense to use it as part of the interview process.

Of course, if you already use pairing stations in your work place then it's likely you also use them during interviews, so kudos to you if you already do, and if you don't actively use pairing stations already, then you should consider trying it out.

3

Inspired by answers to this question:

Most industries have various certifications issued by various authorities. You can save time and effort (and money) while achieving a goal of test-screening candidates by requiring specific relevant certifications.

Invest some time in researching which certifications are most relevant, and which are most meaningful and reliable. Some tests are difficult and thorough, and those holding the certification from that exam are great candidates. Others are easy and less thorough, so having the certification from it might not mean a candidate is highly skilled in that area.

Using this method, you get the benefit of candidates being exam-screened, but without the cost of developing, maintaining, and administering an examination. The only drawback is you don't really have any input with regards to what is tested, so if you are looking for some very specific qualifications, you will have to screen for those directly (phone-screen or in-person interview).

  • 1
    Interesting. Is there any data that people with certifications perform better? I struggle with this a little, as the #1 top programmer I know doesn't have a bachelors degree, let alone any certification. (Then again, he'd never sit through a written exam) – MathAttack Jul 10 '12 at 0:50
  • 1
    I'm very skeptical. Certificates merely indicate that a candidate was able to pass an exam. There will still be enormous variations in performance among the certificate holders. For creative work like programming certificates provide little information. – kevin cline Jul 10 '12 at 15:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.