8

Probably, every software engineer encountered whiteboard interview. For the sake of question let's say that, whiteboard interview is the interview, where candidate is given a problem to solve on a whiteboard, without using actual development tools. And the problem's topic is most probably algorithms and data structures.

If you search the internet for 'whiteboard interview', it seems, that concept of it is seen as something negative. The main reason seems to be, that people think, that result of whiteboard interview doesn't resemble work environment. Thus can't show candidate's ability to get the job done. And it is seen as unfair by both sides of hiring process: 1 2 3.

But all these articles are based on experiences of one or several people. So is there any scientific research on whiteboard interview effectiveness? I don't expect scientific research to answer question 'Are whiteboard interviews bad?'. But I'd expect something along the lines of

We think whiteboard interviews are good to understand X about candidate. We measure "X's goodness" using metric Y. And we measure interview performance using Z. Here is Y(Z) dependence, from which we conclude, assuming Y is good metrics of X, that whiteboard interviews are good/bad enough for understanding X.

It is a tough problem of choosing X and Y. But I suspect, that smart science guys can still come up with some model.

For example. Maybe X could be person's effectiveness, which is really hard to measure, but big companies still try to do it with performance reviews. Let's say Y is the average of first three review grades. And Z will be average of reviewers grades. Is there any correlation between Y and Z.

Maybe even something simpler. Let's answer question "How well do our reviews measure person's involvement in our company?". Then try to measure it with Y - *number of people, who left company during in first 3/6/9/12 month. And draw histogram depending on Z - review grade.

closed as too broad by Dukeling, simbabque, gnat, Michael Grubey, Myles Jul 2 '18 at 13:38

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • What do you plan to do with this research data? – Masked Man Jul 1 '18 at 2:54
  • Nothing particular. I just would like to see whether negative experience of developers backed-up scientifically. – DoctorMoisha Jul 1 '18 at 5:56
  • 1
    @Fattie: Arguably, "research" on this topic would generally be a statistical collection of opinions. This can't be empirically proven, the whiteboard interview isn't proven to be wrong; but it simply misses the main metric by which you measure a developer's aptitude. Asking for answers here is arguably doing the research, as the answers will predominantly be given by either software developers or people who work in a field that's closely related to it. – Flater Jul 2 '18 at 11:14
  • 1
    Whiteboard interview is too poorly defined a concept to have any sort of meaningful studies made on it. Does any interview that involves use of a whiteboard qualify? Only those which involve writing compilable code or those which include conceptual description of solving the problem? If it's conceptual description then does the dataset include other peripherally related fields? It would be a nightmare to define this problem well enough for results to be meaningful. – Myles Jul 2 '18 at 13:44
  • 1
    @DoctorMoisha: Many companies have standards on interview processes. Having a certain standard does not in any way make the outcome meaningfully measurable. The fact that companies have their own standards strongly suggests that there is no objective measure of what is objectively best. So, it wouldn't be so hard inside one company. You are still missing the point that it's impossible to correctly attribute hiring someone (or not) to particular causes such as the whiteboard test. Interviews are inherently multifaceted and subjective evaluations. – Flater Jul 2 '18 at 13:53
1

So is there any scientific research on whiteboard interview effectiveness? There is a tough problem of measuring person's effectiveness, but anything with actual numbers, statistics and conclusions would be great.

How do you measure the effectiveness of an interview?

  • Employer happiness with employee who took an offer?
  • Likelihood that an employee will accept an offer after having done a whiteboard interview?
  • Percentage of applicants who pass the whiteboard exercise?

I know you mentioned the difficulty of evaluation effectiveness, but you then still ask the same question, which seems weird. If you can't objectively evaluate it, then you can't conclusively answer that question.

Maybe some big company calculated true positive rate of people, who passed whiteboard interview successfully and actually do great work.

The assumption that this is a correct metric, is the exact same flawed assumption that leads to thinking whiteboard interviews are a good marker for aptitude.

The issue doesn't lie with hiring people who pass the whiteboard interview (and may or may not be appropriate for the job position). The issue lies with not hiring people who are appropriate for the job position but simply fail to pass a test that does not test their relevant aptitude.

Think of it this way:

An interviewer refuses to hire anyone who spent less than five years in college. The goal of the interviewer is clear: weeding out the "unskilled" applicants, so that only "skilled" applicants are evaluated. However, his metric (time spent in college) is flawed.
I use "skilled" and "unskilled" as an oversimplified discriminator to the interviewer's opinion of the needed skills for the position. I'm not trying to label people.

  • Two types of people will pass the test:
    • Those who achieved a master's degree (skilled)
    • Those who had to often redo courses and eventually managed to get their bachelor degree, almost by brute force and sufficient funds for tuition (unskilled)
  • Two types of people will fail the test:
    • Those who quit studying halfway through (unskilled)
    • Those who were so smart they completed their curriculum in half the time (skilled).

This is why the metric is flawed: it doesn't actually separate the "skilled" from the "unskilled". It does not actually achieve the goal that it is implement to achieve.

The issue with whiteboard interviews is the same. It is used to weed out people who lack certain skills, but it tests the wrong metric and therefore leads interviewers to dismiss applicants with the appropriate skill, as well as potentially offering jobs to applicants based on the wrong skill.

To sum it up in a quote (that is often wrongly attributed to Einstein):

If you judge a fish by its ability to climb a tree, it will live its whole life believing that it is stupid.


The big unmentioned factor here is what the interviewer expects to see on the whiteboard.

  • If they expect to see code with perfect syntax; that's simply an unreal expectation.
  • If they expect to see a basic visual representation of e.g. how a linked list is different from an array; that is something that can meaningfully be explained using a whiteboard.

The main reason seems to be, that people think, that result of whiteboard interview doesn't resemble work environment.

I, for example, am really bad with remembering syntax by heart. However, I am really quick at googling related topics, finding an example snippet, and tailoring it to my needs. I know this is tooting my own horn, but I can outperform most colleagues in terms of speed of implementing something from scratch.

However, if you put me in front of a whiteboard and ask me to write the code for deserializing an XML file, then I wouldn't even know the class used for (de)serialization.
This may lead interviewers to conclude that I don't even know the basics and my coding repertoire is too small; whereas the opposite can be true as well: my coding repertoire is too big to remember all syntax by heart, and I know more than just the basics to a point of simply not focusing on the basics.

This is why "not resembling the work environment" defeats the value of the whiteboard exercise. Not everyone works the same way. Reciting knowledge by heart is not the primary metric by which you measure a developer's aptitude. And that is the crux of why the whiteboard interview is not a good approach.

If anything, you should be measuring a developer's ability to adapt to new systems and to understand existing code, as that is a skill that is much more valuable.
I was given this type of interview once. They had printed out their derived DbContext implementation, and asked me to point out what they implemented. They were looking for answers like soft deleting, pagination, auditing fields, change tracking. They had also introduced two bugs. My ability to evaluate the existing code is much more meaningful than being able to create my own class out of thin air.


Secondly, it leaves the door open to a logical inversion fallacy.

Let's say the interviewer asks you to recite all the methods of Entity Framework by heart, ordered by length of method name (I know it's a silly example, but I want to use an example where we don't get distracted by arguing what the best answer is).

If the applicant is able to do this, then that is (indirect) proof that he must have a lot of experience with Entity Framework. Without Entity Framework experience, he wouldn't be able to do that.

However, it's likely that the interviewer will wrongly invert that logic. If the application is not able to do this, that doesn't mean that he doesn't have any experience with Entity Framework. This inversion is a logical fallacy.

This is why these question are bad. When the applicant is faced with a question like this, he is stuck between a rock and a hard place:

  • The applicant doesn't know if the interviewer has fallen prey to the logical inversion fallacy.
  • If the applicant speaks up about the fallacy, and the interviewer disagrees, the applicant risks hurting his chances of getting an offer.
  • If the applicant speaks up about the fallacy, and the interviewer wasn't falling prey to the fallacy, the applicant could have hurt his chances because he assumed the interviewer was making a mistake when he wasn't.
  • If the applicant therefore assumes that it's better to not being up the fallacy, and he simply fails to do the requested task (which is not a sign of being a bad programmer), then he risks the interviewer assuming the applicant isn't skilled enough for the position.

Regardless of whether the interviewer falls into the fallacy trap or not, the existence of the question always puts the applicant in an awkward position of having to evaluate the interviewer's knowledge (about the fallacy) before he can give an appropriate answer. That's effectively a gamble, and you really shouldn't (not) hire people based on (not) having guessed correctly.

  • I did say "I agree" for that part. The first point of my comment was, with the witeboard interview question like "what is a linked chain", you may test at best one skill you really need from the guy (or 0 at worst), but not another like handling the client where the hiring manager wants him to do such stuff. This means for each skill you want to test, you have to prepare an adequate question, which is not so easy. I do feel like it is said thorough the whole post of course, but I though it could deserve its own bold sentence or whatever :) – Walfrat Jul 2 '18 at 13:01
  • Sorry, I don't like your answer, because it doesn't answer my question. If I understood you correctly - you say, that if something is difficult to measure - it is not worth to measure it at all. But I don't think it is true. Yes, measuring employee contribution is hard and flawed. Still, many companies do performance reviews, grading employee's contribution. This can't answer the question - is whiteboard interview good or bad, but gives insights on impact of whiteboard interviews. – DoctorMoisha Jul 2 '18 at 13:09
  • @DoctorMoisha: If I understood you correctly - you say, that if something is difficult to measure - it is not worth to measure it at all. That's not what I'm saying. What I'm saying is that it's impossible to attribute the success of an interview to a single cause. You're trying to use approximate metrics that are not precise enough to give you a meainingful answer (which is similar to the flaw with whiteboard interviews: overgeneralized metrics that don't touch on the core focus). This is also wholly different from performance reviews or grading contribution. – Flater Jul 2 '18 at 13:13
  • @DoctorMoisha: Performance reviews (and contributions) are assessing the employee in their natural working environment. Whiteboard interviews are assessing the employee outside of their natural working environment. That is the key difference, and the core argument as to why whiteboard interviews are not as relevant as interviewers think they are. Whiteboard interviews also take place in too short a timeframe (10-20 minutes) to meaningfully assess an employee's skills, whereas performance reviews take a long timeframe (3 months to 1 year) – Flater Jul 2 '18 at 13:18
  • @DoctorMoisha Sorry, I don't like your answer, because it doesn't answer my question. It does answer your question, the answer is simply not what you're expecting. If you're asking if something exists, and also exclude the possibility that it doesn't exist, then your question is moot. I can't prove a negative (the absence of the existence of research). If you wish to assert that research exists, then the onus is on you to prove your point. – Flater Jul 2 '18 at 13:22

Not the answer you're looking for? Browse other questions tagged or ask your own question.