Testing as part of the interview process

Question

I am curious what people's thoughts are on various types of testing in the interview process.

For the sake of argument, let's start with the following as given:

1. Interviews are a very sub-par way to determine good employees from bad. (It can be hard to tell if someone is worth hiring after a 10 week internship. After a 1 hour interview it is nearly impossible)

2. Interviews are expensive if you haven't pre-screened well.

3. There is a lot of bias in the interview process.

4. With grade inflation, academic credentials aren't what they used to be.

5. If you don't know the person doing a reference, it likely isn't useful.

So my big question is - what types of testing can be worthwhile (and legal) in this context?

More specifically: A lot of people do whiteboard coding for programming interviews, but that doesn't solve #2 and #3. Can a written exam given to a larger subset of people help? Are there HR/legal issues to be worked around? Also - are there other "subset of a job" techniques that people have used in interviewing? (Make a presentation, show me something you've written, etc.)

For the sake of guiding the answers, can you comment on thoughts specific to written exams or other non-standard interviewing techniques? Thanks!

Answer 1

Interviewing is more art than science. Some people are not good test takers but thrive under real life pressure. Others are great test takers but fold in real world conditions.

The most important thing is to find a method that works for your group.

The most important thing is to get the right number of candidates to each phase screening process. If you get too few to any one phase then you start feeling pressure to lower standards and expectations to fill positions. This often leads to poorly qualified people being over their heads and unable to perform. The result is a negative for both sides. If you get to many then it is difficult to make a good choice and you end up turning away great candidates many times unrealized. What is the right number is a subjective matter that varies. You need to find the right numbers for your organization

Most commonly I have seen a 4 phase process.

Prescreen: An initial check to make sure the applicant meets the bare minimum requirements. There is no point in interviewing someone that can not be brought on anyway. Some companies have used an automated test here as well.

Phone Screen: The initial interaction with the company, simple tech test, introduction to the company, verification of expectations and next steps.

Skill Test: Gauge the skill level of the candidate. I have seen the range of FizzBuzz to a Kobayashi Maru type(there was no way to actually solve it just about how you handle the test). Personally I dislike the unbeatable test because I think it sets a bad tone for the relationship. I prefer that candidates come on board with a positive experience of success rather than failure but that is my preference.

Team Interview This is a test of the culture fit. This is not so much about what they know but how they interact with the key people on the team. Technical questions are asked not to probe knowledge so much as to gauge levels of interest and compatibility.

Hiring is expensive. Made more so if you hire to many people who do not work out. There is no golden bullet but getting the right number of people to each phase is important to keep your team/business healthy.

Answer 2

Pre-Evaluation: Ask Them to Rate their Skills

I ask applicants to rate themselves for all the technologies on their resumes according to this scale:

 1 - Read the book on X.
 2 - Reviewed the book on X.
 3 - Wrote *THE* book on X.

Make it very clear that this is to be taken very close to litteraly.

Inevitably, a majority of people will send back evaluations with at leat one 3 and a few 2, but not a majority one 1s.

Of course, you can't determine and evaluate someone's skills based on these raw numbers and their own evaluation. But it establishes a trust contract, and many are keen to break it. Or incapable to read a simple request and stick to it. When you do your phone screening, keep this evaluation handy, and if after 2 questions they are obviously one level down in most of their key technologies, they were overestimating themselves. And if they're below the level of acceptance you've set for the role, just call it a day. There's no guarantee they could be still good people, but you have better things to do.

It's a nice approach, as it forces people to really reflect about their skills. The good ones are the ones who would have put their level as "advanced" on their resume, and suddenly realize what "advanced" really should mean, and do have humility to backtrack.

The issue with this approach is agencies. They are usually annoyed by this, as they make their money from finding people, and you just lengthen their process, which may already be long, and basically might scare off some people.

Pre-Test: Be Challenging and Strict

I don't mind giving pre-screening exercises. It's controversial as not everybody will have the time to conduct them, and some can cheat on them, but you just had some barriers to resolution. For programming roles, it's especially easy to do as you can request applicants to submit programs that can be marked by a program or evaluated quickly. You'd need to write these programs once, and then you're done. It's a good time investment.

Then send out these exercises, and make sure the guidelines are crystal clear on what output you expect. Rigor is something of great value with developers, so you want to find the ones who CAN READ A BASIC ASSIGNMENT AND IMPLEMENT IT WITHOUT WHINING. Writing small programs that return some numbers or strings and comply to the assignment exactly (no EXTRA characters here and there) is easy.

Just with that, I often weed out non-motivated applicants, or some take offense. It's fine. It's communication. They're basically telling you, knowingly or not, that they weren't that interested. You just saved time on the next interviews.

For the ones who send something back, if some send exactly what you want, be sure to keep a close eye on these. They aren't that common, though they should be. For the ones who fail entirely, you can probably dismiss them. For the ones who failed to follow the guidelines by a small margin, still get them through, if you can: it's just one more rating point you can use in your decision taking process later, even if it's not entirely conclusive at the moment.

These need to be simple (in terms of implementation and time it takes) assignments, but with a problem-solving touch.

Phone Screening: Be More Flexible, Cover A Lot of Ground

If this is for a short phone screen, put them at ease (the usual yadda-yadda: "drive us through your resume with emphasis on what you think matters for this role", "what are you looking for", and describe a bit more about the role and ask if they have questions.

Don't make it last too long, as otherwise they're getting nervously tired, so get it out of the way quick, and start the phone tests.

Ask clear, short questions, and make sure to be more flexible than for pre-tests. They are nervous, so give them time, and lead them towards the solution. You want to detect a "mindset" here, and a "personality", not raw skills.

If you have time for a long-ish phone screen, use online editors with collaborative capabilities (EtherPad, Google Docs...), and help them through problems. Keep backups of these for the later stages to refresh your memory about applicants and be able to review all of them once you've seen a few: you might revise your judgment based on the individuals you've rated so far. Some EtherPad clones even allow to record the session, so you can do a play by play of the session's typing, which can be very interesting. When you ask a question verbally, make sure to have it already pre-written in another editor to copy-paste it (and possible code stubs, comments and examples) in the shared session to not waste time.

On-Site Test: The Employee-Experience

For face-to-face interview(s), you want the whole shebang:

• resume review,
• whiteboard brainstorms,
• technology-centric discussions,
• informal chats (around water-coolers or even over lunch),
• hands-on tests.

All of these are rather self-explanatory, but the last one has some specifics: do have a computer ready with pre-installed tools, and assignments. Make sure to have the capability to later record the applicant's work. I'm not saying to live record, but just sending the coding assignment to a version control system or document management system is great for future reviews, especially if candidates re-apply later.

Involve multiple aspects of the daily job in the coding test, and have some paper-based and computer-based components:

• paper-based code-review,
• live code-review (by rewriting the code, or annotating it in a Crucible or ReviewBoard if you have one with a guest account),
• live coding-test (make sure that this includes writing code AND writing tests for the code. Or writing test for already existing code. Or writing code for existing tests.)

Debrief quickly with them so they can run you through it, ask for final closing statements and questions, and call it a day.

Make sure that multiple team members take part to the tests and interviews, possibly some senior developers and some junior developers or even interns (great training for them), and even people from other departments (QA, training, support ...), if only for a few minutes. You'll all end up working together. So while it's important you to have the final decision, it's good for other to be able to weigh. Make sure you rotate these panels of applicants during the different phases of the interview, with some elements staying throughout: this way you can avoid possible conflicts with the odd employee who really can't stay a guy (though, in the end, if that were the case, you unfortunately probably should pass even on a good applicant).

Obviously, I rarely interview more than 3 applicants per day. More likely 2: one in the morning, one in the afternoon, or make them overlap in the way that's not annoying.

---

How to Keep it Cheap

All this seems considerably time-consuming. And it is, and you want to invest some time in finding long-term hires, for a great ROI. But, if you automate most of this work, and make it part of your process, it takes away a lot of the routine and guesswork, and you can get pretty efficient at it. Of course, at some point it even gets a bit robotic, which isn't too good, but it's hard to avoid.

The pre-evaluations and pre-tests seem harsh and like they could weed out or scare away very good potential candidates, and that's probably the case. Maybe some of these who give up would have been great hires if they had gone through. But the cost of dealing with all the other ones who wouldn't have been great hires is far more threatening and depressing thatn the perspective of not seeing a stellar employee walk through the door. And the process aims at finding you other stellar employees. So don't think of those giving-ups as lost potential hires, but as time-savers, giving up space for the next big ones.

Answer 3

For tests on written work I would focus on:

• Their code samples. Go through the examples they provide with them and talk about the approach and the details.

• Their network. See who their peers (via linkedIn, Stack Overflow, etc. are) and what kinda reputation they have for code.

• Whiteboard development of some simple code. I generally prefer to stay away from much hands-on machine work as people can be nervous on the keyboard and/or unfamiliar with tools you want them to use. Smart folks will learn the whatever the setup is, so focus on their approach for screening.

Answer 4

From what I heard a very good technique of testing potential programmer employees is to schedule some time for video conference and ask them to solve some simple problems in a specific amount of time. This approach doesn't take too much time (and can be streamlined with some automatic tools if there are a lot of interviews), and allows the interviewer to screen people that don't have the right skills before the proper interview.

As far as I remember the technique was used by some respectable company and the results were quite astonishing - a lot of programmers applying for their positions failed such basic tests. This allowed them to weed out a lot of candidates and leave room for more proper interviews for the ones that passed this "bare essentials" test.

Answer 5

Are you considering behavioral interviewing in your process? Research appears to support its efficacy over theoretical test questions, especially for high-level jobs; see for example Motowidlo et al., Pulakos and Schmitt (summarized by Huffcutt et al.). The former article correlates information from recorded reports (not just live interviews) with judgment validity. I believe this would apply to written behavioral interview tests, although I have only practiced behavior interviewing in person. There are probably established guidelines for a written approach if you choose to take it. If you go with more of an in-person route, the research suggests interviewer skill is an important factor.

Answer 6

A way test people and find out if people are going to fit with your team is to involve the candidate in the real process that you have with your team instead of inventing scenarios just for the interview or coming up with trick questions that supposedly tests the knowledge that they have. This makes for a better team interview, because unless your team interviews people often it turns out they are probably ill-prepared and very bad at interviewing people! Afterall, do you sit around with your team on a regular basis, singling out one person to ask them random technical trivia questions? That's the way most team interviews are conducted and it feels alien and isolating and nothing like the actual working environment. So, make it real...

During the later stages of the interview process treat them as if you hired them as a consultant to work on a specific problem for part of a day. At the later stages of the interview process both you and they should be willing to commit a good 2- to 4-hour (or more!) chunk of time to the interview process. That should be sufficient time to involve them in a real meeting with the team members they would be working with to discuss and begin working on an actual problem your team is trying to solve. Yes, that takes quite a bit more preparation and planning on your side, but it will tell you a lot more about how someone thinks and works along side others than any number of interview questions you might ask.

The hard part of this is carving out a well-defined problem that is solvable, at least in part, in a reasonable amount of time, that is deep enough to touch on multiple technical areas, and that requires enough interaction with multiple members of your team to get a sense of how they approach problems and work with others to solve them.

Working along side others is much better than isolating them in a room with a problem to solve since their communication skills and ability to work with your team is often more valuable than any domain specific knowledge they may bring with them; not that the latter isn't important, but it's not worth hiring the smartest person in the world if they're gonna act like a jerk and can't work with everyone else.

Something that makes this process easier is if you already use pairing stations with your current team, as is popular these days in some work places. Note I said pairing station rather than pair programming station, because a pair doesn't have to be limited to just writing code-- other office disciplines can benefit from that kind of setup, as well. I don't recommend using a pairing station just for the interview, though! If you don't already use them in your workplace with your team, then it will be another artificial part of the interview. For the team that I manage we don't use pairing stations for everything, but it is part of our office setup and I make sure that we make use of them when it is effective to use them, so in our environment it makes sense to use it as part of the interview process.

Of course, if you already use pairing stations in your work place then it's likely you also use them during interviews, so kudos to you if you already do, and if you don't actively use pairing stations already, then you should consider trying it out.

Answer 7

Inspired by answers to this question:

Most industries have various certifications issued by various authorities. You can save time and effort (and money) while achieving a goal of test-screening candidates by requiring specific relevant certifications.

Invest some time in researching which certifications are most relevant, and which are most meaningful and reliable. Some tests are difficult and thorough, and those holding the certification from that exam are great candidates. Others are easy and less thorough, so having the certification from it might not mean a candidate is highly skilled in that area.

Using this method, you get the benefit of candidates being exam-screened, but without the cost of developing, maintaining, and administering an examination. The only drawback is you don't really have any input with regards to what is tested, so if you are looking for some very specific qualifications, you will have to screen for those directly (phone-screen or in-person interview).