How can one mitigate cheating on a pre-interview technical test?

Question

One strategy to screen out inappropriate candidates in a technical field is to give them a programming test, and only invite candidates who perform well.

For example, if I wanted to employ a programmer, I might send them a test from codility.com.

My question is:

Without supervising the entire test by webcam, how can I know that the candidate who completed the test is the same person applying for the vacancy?

Similarly, are there other ways of cheating this test, and can they be mitigated?

Obviously, we can weed out the non-programmers at the next stage - if they can't support the answers on the test, obviously it wasn't them - but I'd like not to have to waste the time phone-interviewing a possible cheater in the first place

This question is somewhat related to: Preventing cheating in a phone interview, but the notable difference is that I'm definitely not asking about cheating on trivia-type questions here, I'm referring to in-depth programming knowledge. Given the nature of a remote programming test, Googling for answer strategies is acceptable and encouraged.

Answer 1

I don't believe you can come up with a control that is both reasonably effective and less time consuming than bringing the candidate in and verifying that they can support answers.

My approach would be to make the stakes high enough that cheating is also shooting yourself in the foot. Have the candidates sign off on an honor code that clearly states that if they are found cheating, both the person who showed up for the interview and the name & contact details of the resume provided will be blacklisted from company searches for 5 years. It's reasonably easy to keep a listing of cheaters - their pictures, their aliases, and their phone numbers and email addresses. Then use that as criteria #1 in screening future candidates.

Yes - it's still possible to cheat your way around that - new phone number, new alias, etc. But at some point, your ability to weed out people with no skills will be clear enough that it doesn't seem worth the work.

In all honesty, if this is happening for any significant percentage of the time - you may have to give up on the pretest. And if it isn't a significantly frequent occurrence, figure the time wasted on verifying the candidate in person is going to be more minimal than all the tech work of doing more.

Answer 2

Just as the important thing about a whiteboard programming test is the conversation it facilitates, the important thing about a take home test can be the conversation it facilitates.

Tell them to keep the code handy because there will be a conversation about it afterwards. Then, for those whose submissions are good enough for a phone or in-person interview, you can ask stuff:

• did you write these tests before or after the UpdateTotals() method? why?
• you're using [tool whose artifacts are in their submission - eg .sln means Visual Studio] - what do you like about it?
• have you run [static analysis, code prettying, etc] on this code?
• what warning level did you build this at? why?
• You chose a loop rather than recursion, why?
• Is NewPolicy() thread-safe? Should it be? What would need to change? Is that worth it?

Someone who didn't write the code will bomb these questions totally. Someone who did might still do poorly. You will have a framework and a context for the sorts of questions people love to ask in interviews, while also proving they did the assignment themselves.

(A really suspicious or cruel interviewer could ask trick questions, like asking about something that isn't even there, but I wouldn't.)

Answer 3

Without webcam-invigilating the entire test, how can I know that the candidate who completed the test is the same person applying for the vacancy?

If you simply send off a take-home exam and later receive the results, then the answer is you cannot know for sure that the test taker and the candidate who shows up in person are actually the same individual.

There are organizations that will proctor a test for you, and check identification before the exam begins. If having someone else "take the test" is that much of a repeated problem for you, you might have to pay to have it solved or have folks take the test in your office, rather than remotely.

Although I could imagine it happening, I haven't actually seen this as a real problem in my companies. But perhaps your context is different, perhaps you are hiring in volumes that make even a 15 minute phone screen prohibited, or perhaps other issues will get in the way.

Answer 4

In a sense, you are looking for a way to verify that the interviewee is passing some kind of Turing Test. For example, even if the interviewee visits your office in person, she or he could have a subcutaneous radio transmitter embedded in their ear canal that is hooked up to Watson over at IBM. Are you planning to check for subcutaneous ear transmitters for all the candidates who visit you? No, because the likelihood of that kind of cheating is low, and the cost to the cheater is high enough that few cheaters would be willing to bother.

So the goal is to orient the test so that cheating is expensive, and so that the signal you receive from the candidate is hard-to-fake. Nothing is ever perfectly cheat-proof, so you need to decide on how expensive you want to make the evaluation. A test that's ridiculously elaborate will drive away potentially good candidates who don't have the time to putz around with complicated tests in their already busy lives. A test that's too easy will get many good submissions from many folks, making it hard to separate the sheep and goats.

In the domain of programming, some tips that I have used when building and conducting programming tests:

1. Make the test less about trivia and more about creativity / ingenuity. For example, I never explicitly ask people to submit unit tests with their code samples, but I penalize them heavily if they don't simply do that all on their own. If they take the initiative to make something good, rather than something that minimally meets the requirements of the test, it's less likely they are cheating (since cheating costs someone else to use time, they are less likely to go the extra mile).
2. Make the question very open-ended, allow the candidate to use Google or other people, but ask for an explanation for their approach. By the time the "cheater" has explained all about the cheating approach taken, the candidate ought to actually know about it, well enough to write a write-up. Here again, unless the cheater has access to a magic oracle person with no personal time constraints, it's going to be expensive to get someone else to take a lot of time to both solve a problem creatively and write the write-up.
3. Use tools like collabedit or Google Docs to have the candidate solve a problem in real time with one of your programmers. During the interview, make sure to ask questions that are very specific, and look for quick answers. If the programmer is doing well at solving the problem, they should be able to de-focus on the programming, quickly answer related side questions, and then re-focus on the programming. If they need extended pauses every single time to answer routine side questions, they are probably consulting something else for help.
4. Make the candidate pass through several filters. Ask a wide range of questions that have a wide range of difficulty. After all, a big part of technical interviewing is to figure out the technical skill boundaries of a candidate. No one knows everything and you need to know where the limits, cracks, and low aptitude areas are. If you give a phone screen with 5 questions, ranging from easy to hard, document both the candidate's answers and also the speed with which they answer. Then give another phone screen that is similar, and possibly a third. It will be difficult for a candidate who is cheating to arrange for the same assistant to be there all the time, and if you see glaring inconsistencies in aptitude on certain kinds of questions, it can be a red flag.
5. Ask for a certified sample instead of the answer to a test. Ask the candidate to submit a sample of code from a college course, an open source project, etc. and ask for the name of a reference (not a student, but a faculty member, a teacher, a teaching assistant, a project leader) who can vouch that the candidate wrote the code. Then ask the candidate to explain their coding choices for that problem / project.

The basic idea here is that (1) make cheating expensive and (2) find a trade-off for the expensiveness of (1) such that good non-cheaters are not put off by the barrier to entry at your company.

I would also add that in my years of experience designing and conducting programming tests at my current company, cheating is the least of my worries. Few candidates can even follow basic instructions. For example, throw in a simple instruction such as: "You are not allowed to use built-in string manipulation functions to solve this problem." In my experience, greater than 50% of the candidates still use built-in functions, unapologetically, and have no answer when asked why they did it.

If you just use a few of these screens, it will beat out a large percentage of cheating, laziness, and low aptitude noise. Then, those few extremely diligent cheaters who make it through will be less of a time-waste for your company.

You might also try telling the candidates that they may receive help from a friend, but only if they tell you the names of all the people who helped them. If the candidate doesn't seem like the genuine article in the in-person interview, then hey, maybe you can call up their friend who helped them cheat and gave the better answer in the coding test?

Answer 5

I wouldn't worry about cheating at this point at all.

You will have lots of potential candidates, and you want to filter out those that are useless. The test filters out those that are useless, except the cheaters. So don't take this test as evidence of any ability, just as the key to coming to the in person interview.

You could as the first question in the interview ask about the test. My current company used codility.com (I think) and a cheater would have been completely, absolutely, stuck when asked about one of the problems posed (which was a very cleverly hidden logic bug).

The idea is that the test is for filtering out people where an interview is a waste of your time. Cheaters will be able to waste your time, but they won't be able to get a job. If you are really worried about cheaters, take a photo of each candidate during the interview because I know of cases where the person starting the job was not the person doing the interview.

Answer 6

Your goal with a programming test, like an interview or any other hiring process, is to help determine if someone is a good fit for the company. If you need verification that they actually took the test, (and assuming you can control the contents of the test) you can accept the test as-is and bounce off it in future interviews.

For instance, you could ask in a test to develop a simple program that requires a couple different data structures in a language where there are several classes to fit that bill, then ask in an interview why they chose that one over another. If they gave you "perfect" code that implies strong competency with the language, then can't tell you why they chose an StringBuilder over a String or some other type, then you probably have your red flag.

Answer 7

Two other options not listed above are:

1. Have the requirements vague. Mention you need to compare strings, but don't mention if the comparison needs to be case or language sensitive. Use dates and times, and see if the programmer asks about time zones and leap years. This gives you an idea that the person knows enough to ask the right questions -- if they go straight to a Google search, you know that they might not be the right person.

2. Ask them to tweak the program during the interview. Tell them, "the requirements changed -- the end user asked if you can give them this extra information. How would you alter your code to provide this?" You could tell that some people didn't feel comfortable maintaining the code they submitted to you (which strongly suggests that someone did the work for them or they found it online and didn't totally understand the code).

Answer 8

This may not be exactly the answer that you're looking for, but I handle this by giving developer candidates a simple 10-question quiz at the beginning of an in-person interview. The questions are geared toward the expected skill level for the position and cover concepts, not code samples. For example, I may ask a junior .NET developer candidate to describe the difference between a method and property, or to explain what LINQ does and how it's an improvement over its predecessors. I may ask a .NET architect candidate to explain boxing, its impact, and how it can be mitigated.

This is obviously not as good as a programming assignment, but it does give me a reasonably good view of the candidate's knowledge of the concepts and technologies that he/she will be using. It also gives me good insight into their written communications skills as well.

If the candidate survives this test, I expect any deeper questions about technique, etc to shake out in the second interview with the development team.

Answer 9

Ask questions that don't have canned answers, let me give you an example:

1. What is the difference between a class and an interface? Google has everything you need to answer this, so you're testing memorization here, not skill.

2. When would you use a class vs an interface? Much more interesting answers that demonstrate skill, experience, and understanding.

A technical test full of 2. type questions would be much harder to cheat on and give a better assessment of the candidate's skills.