Pre-Evaluation: Ask Them to Rate their Skills
I ask applicants to rate themselves for all the technologies on their resumes according to this scale:
1 - Read the book on X.
2 - Reviewed the book on X.
3 - Wrote *THE* book on X.
Make it very clear that this is to be taken very close to litteraly.
Inevitably, a majority of people will send back evaluations with at leat one 3 and a few 2, but not a majority one 1s.
Of course, you can't determine and evaluate someone's skills based on these raw numbers and their own evaluation. But it establishes a trust contract, and many are keen to break it. Or incapable to read a simple request and stick to it. When you do your phone screening, keep this evaluation handy, and if after 2 questions they are obviously one level down in most of their key technologies, they were overestimating themselves. And if they're below the level of acceptance you've set for the role, just call it a day. There's no guarantee they could be still good people, but you have better things to do.
It's a nice approach, as it forces people to really reflect about their skills. The good ones are the ones who would have put their level as "advanced" on their resume, and suddenly realize what "advanced" really should mean, and do have humility to backtrack.
The issue with this approach is agencies. They are usually annoyed by this, as they make their money from finding people, and you just lengthen their process, which may already be long, and basically might scare off some people.
Pre-Test: Be Challenging and Strict
I don't mind giving pre-screening exercises. It's controversial as not everybody will have the time to conduct them, and some can cheat on them, but you just had some barriers to resolution. For programming roles, it's especially easy to do as you can request applicants to submit programs that can be marked by a program or evaluated quickly. You'd need to write these programs once, and then you're done. It's a good time investment.
Then send out these exercises, and make sure the guidelines are crystal clear on what output you expect. Rigor is something of great value with developers, so you want to find the ones who CAN READ A BASIC ASSIGNMENT AND IMPLEMENT IT WITHOUT WHINING. Writing small programs that return some numbers or strings and comply to the assignment exactly (no EXTRA characters here and there) is easy.
Just with that, I often weed out non-motivated applicants, or some take offense. It's fine. It's communication. They're basically telling you, knowingly or not, that they weren't that interested. You just saved time on the next interviews.
For the ones who send something back, if some send exactly what you want, be sure to keep a close eye on these. They aren't that common, though they should be. For the ones who fail entirely, you can probably dismiss them. For the ones who failed to follow the guidelines by a small margin, still get them through, if you can: it's just one more rating point you can use in your decision taking process later, even if it's not entirely conclusive at the moment.
These need to be simple (in terms of implementation and time it takes) assignments, but with a problem-solving touch.
Phone Screening: Be More Flexible, Cover A Lot of Ground
If this is for a short phone screen, put them at ease (the usual yadda-yadda: "drive us through your resume with emphasis on what you think matters for this role", "what are you looking for", and describe a bit more about the role and ask if they have questions.
Don't make it last too long, as otherwise they're getting nervously tired, so get it out of the way quick, and start the phone tests.
Ask clear, short questions, and make sure to be more flexible than for pre-tests. They are nervous, so give them time, and lead them towards the solution. You want to detect a "mindset" here, and a "personality", not raw skills.
If you have time for a long-ish phone screen, use online editors with collaborative capabilities (EtherPad, Google Docs...), and help them through problems. Keep backups of these for the later stages to refresh your memory about applicants and be able to review all of them once you've seen a few: you might revise your judgment based on the individuals you've rated so far. Some EtherPad clones even allow to record the session, so you can do a play by play of the session's typing, which can be very interesting. When you ask a question verbally, make sure to have it already pre-written in another editor to copy-paste it (and possible code stubs, comments and examples) in the shared session to not waste time.
On-Site Test: The Employee-Experience
For face-to-face interview(s), you want the whole shebang:
- resume review,
- whiteboard brainstorms,
- technology-centric discussions,
- informal chats (around water-coolers or even over lunch),
- hands-on tests.
All of these are rather self-explanatory, but the last one has some specifics: do have a computer ready with pre-installed tools, and assignments. Make sure to have the capability to later record the applicant's work. I'm not saying to live record, but just sending the coding assignment to a version control system or document management system is great for future reviews, especially if candidates re-apply later.
Involve multiple aspects of the daily job in the coding test, and have some paper-based and computer-based components:
- paper-based code-review,
- live code-review (by rewriting the code, or annotating it in a Crucible or ReviewBoard if you have one with a guest account),
- live coding-test (make sure that this includes writing code AND writing tests for the code. Or writing test for already existing code. Or writing code for existing tests.)
Debrief quickly with them so they can run you through it, ask for final closing statements and questions, and call it a day.
Make sure that multiple team members take part to the tests and interviews, possibly some senior developers and some junior developers or even interns (great training for them), and even people from other departments (QA, training, support ...), if only for a few minutes. You'll all end up working together. So while it's important you to have the final decision, it's good for other to be able to weigh. Make sure you rotate these panels of applicants during the different phases of the interview, with some elements staying throughout: this way you can avoid possible conflicts with the odd employee who really can't stay a guy (though, in the end, if that were the case, you unfortunately probably should pass even on a good applicant).
Obviously, I rarely interview more than 3 applicants per day. More likely 2: one in the morning, one in the afternoon, or make them overlap in the way that's not annoying.
How to Keep it Cheap
All this seems considerably time-consuming. And it is, and you want to invest some time in finding long-term hires, for a great ROI. But, if you automate most of this work, and make it part of your process, it takes away a lot of the routine and guesswork, and you can get pretty efficient at it. Of course, at some point it even gets a bit robotic, which isn't too good, but it's hard to avoid.
The pre-evaluations and pre-tests seem harsh and like they could weed out or scare away very good potential candidates, and that's probably the case. Maybe some of these who give up would have been great hires if they had gone through. But the cost of dealing with all the other ones who wouldn't have been great hires is far more threatening and depressing thatn the perspective of not seeing a stellar employee walk through the door. And the process aims at finding you other stellar employees. So don't think of those giving-ups as lost potential hires, but as time-savers, giving up space for the next big ones.