A programming task is scaring off candidates, should we ditch it?

Question

It is the first time that I am doing HR, and we are looking for a developer. The selection process is three rounds: technical phone interview, programming task (0.5 - 1 hr challenge), and then finally an interview with upper management and me.

The problem that I am having is that when I give some candidates (mostly fresh grads with 1 or 2 technical internships already under their belt) the programming task, they not only do not complete it in the given time frame (a week), but I don't hear from them again unless I follow up.

I'm thinking about ditching the programming task, but I really think it can help establish who really knows the languages listed on their resume.

How can I improve our hiring process?

UPDATE SINCE THIS QUESTION HAS BEEN POSTED

For the most part a lot of candidates are put off by the programming tests, which is very frustrating since I need to go through A LOT of candidates to find one that is willing to do it. With that said, I have found some willing to do them, and generally have found that:

a) It shows that they have a good attitude towards the company and role if they are prepared to go the extra mile to complete it.

b) They have programming ability. Sure they may have cheated but if they have attempted it, again, a good attitude for us is much more important since they will be much more easily manageable.

I have since hired a graduate developer, who attempted and completed the exercise successfully. He also completed all the additional 'bonus points' on the task. It is too early to say how good he is at his role, since he has only recently started.

On some occasions I do avoid giving the exercise to developers if they already have a strong portfolio working for good brands, and a track record. Since to get into the large brands he would have to do their own entrance tests.

SECOND UPDATE SINCE THIS QUESTION HAS BEEN POSTED

The graduate developer has turned out to be a star employee. We have kept him on and gave him a payrise.

Answer 1

You need a programming test. But it should be 5-10 minutes. 30 minutes tops. There is no 2 hour test that will tell you exactly how good a programmer is. You won't be able to tell if they continually write maintainable code, or if they always comment properly, or if they come up with unreadable messes of 'clever' solutions to problems, etc. In 2 hours, the only thing you'll accomplish is find out who is outright lying about knowing a language/programming.

Except you should be able to spot outright frauds in far less time than 2 hours. 5 minutes writing FizzBuzz and 2-3 quick questions about language specific features will tell you if they are absolutely worthless. And that's about the best you can do in a job interview.

Let me tell you what I would think if I received a 2 hour exam for the privilege of conducting an interview:

"Either these people think this has value which means they have no clue what they are doing, OR, they know this is a waste of time, but for some reason (probably blindly following HR's red tape), they are willing to waste the candidate's time before the candidate is even hired. Imagine what they'll think they can get away with once they are paying me.

Either way, I don't want to work there."

---

---

There's another thing that could be driving away candidates: The length of your interview process. You have people doing a phone interview. Then a take home test that they have a week to finish. Then you go over the tests. Then you set up a face interview with management. Then (I assume) you do a couple of face interviews. Then you get back to the guy who you want to hire. What does that take? 2 weeks? Longer?

In that time I've already gotten 3 offers. I accepted one and I start next week. You think I'm going to go do your 2 hour test?

Answer 2

Let me begin by saying hiring employees in general is a pathetically inexact science, and you will get various results no matter what you try. Having said that, I felt I should share my thoughts on this.

I think programming exercises are a bust, mainly because you will get people who are desperate and have way too much time on their hands completing it. If they have a full time job, you'd have to ask how much time/mental energy they are taking from their current employer to work for someone who isn't even paying them. Do you want them doing things like that to you?

In addition, if they are good, they are most likely interviewing with many employers. Guess which ones they will prioritize? The ones who don't ask to complete projects before they even get on the phone with an engineering manager.

There is also the problem of plagiarism. Sure, they will be found out in the in-person interview, but by then you've already wasted the time of (presumably) highly-paid people at the company interviewing this person.

My current company did it well, and this is the route I would go down in your position. Give them a small task which should only take maybe 90-120 minutes to do, and tell them to justify in the comments why they chose that way of solving the problem. This is something that can be done in one night, and will give you insight to their thinking.

I can tell you right now that if I get a project that takes 8+ hours to do, I tell them thank you but I'm not interested. I have a good job and a life. If a manager sees that as an issue, that tells me they wouldn't care about my work/life balance on the job (especially if they don't care before I have it). No company except maybe Google, Apple, or Facebook could get away with that.

Answer 3

In my experience, take-home projects don't weed out bad coders, and likely make good coders find a job where they don't have to jump through a hoop to talk to the hiring manager.

Think about it this way. A good coder can easily get phone and in-person interviews. Every extra hoop they have to jump through will mean they'll take the easier route and interview (and get hired) by someone else who will pay just as much. If you make people jump through hoops, make sure they see it's worth their while from the get-go.

A bad developer can take as long as they want. You won't see them take 4x as long; you'll just see the completed test. You also won't see them go to Google or their buddy for help with an if statement.

My last company did this, and the vast majority of people we brought in for an in-person interview failed Fizz buzz (around 65%). I think this happened because we inadvertently weeded out good, busy people that didn't need another company's interview, and at the same time, dangling an easy interview in front of bad coders.

What I think we should have done instead

Instead of giving people a take-home assignment that took 15-20 minutes to grade, we should have done a 15-20 minute Skype interview where we asked questions like Fizz buzz. This would have taken the same amount of time, but I would likely have weeded out the bad programmers before a two-hour in-person interview.

FYI -> Here is a very detailed article about Fizz-buzz and general interviewing practices.

Answer 4

A contrasting view, from someone in the trenches on both sides of this question. I suspect this answer will attract down-votes, but it's a highly informed opinion, developed over several decades in this business. Because I like doing so, I still write code every day for a living and as a hobby in my "copious free time", but I've also been the final decision maker on hiring a couple dozen programmers, and have helped to interview and advised on the hiring of a fairly large number more.

Lawrence is right: hiring is woefully inexact. But we're getting better at it over time. More than face-to-face chats, more than trivia questions, short on-site programming challenges are an increasingly essential part of that: not only for skill, but for attitude.

Note that I said "short". We give candidates a laptop set up with a properly installed Eclipse (our chosen IDE) when they come in for the interview. Properly set up means they have access to the jdk source, but it has no internet, and they're asked to not use cell phones during the test. They get an hour to solve anywhere from zero to five problems, starting from "FizzBuzz", and going up in complexity to something on the order of a simple multi-threaded producer/multiple-consumer system.

Nobody's ever solved all five in an hour, and I don't expect anyone ever will (I can write solutions for five in comfortably under an hour, but of course I've seen the problems and solved them all previously, so I'm not a fair test).

Incidentally, we've had ostensibly experienced programmers fail to get FizzBuzz right. Let me say that again: we've had a handful of candidates fail to complete FizzBuzz correctly. Usually by not following directions, but reading the spec is an essential part of being a developer.

We have them write code while here on premises rather than giving them homework for several reasons, partly the risk of cheating, and partly to give each candidate an equal opportunity to work the problems. We also rotate problems in and out of the challenge set, and we discuss solutions with candidates afterwards.

I feel extremely strongly on this issue. I've never seen a candidate refuse a position or drop out of the interview process because of our programming test. But that may be influenced by the fact that we tell the recruiters who serve us that the process includes on-site programming. So we might just not see the candidates who think being asked to write an hour's worth of code is "unpaid work". If so, good riddance. If they think an hour's coding is more like "work" than an hour of answering trivia questions, or that their answers to test problems can produce anything useful, I don't need their attitude.

We did once have one candidate complain about the test, because he thought he was too senior to have to prove himself (that's not speculation - he said so). And he did well on the challenges, and he had all the experience we wanted. But he was a terrible employee: he also apparently thought he was too senior to take direction or learn, and ultimately we had to let him go.

One of the things I've decided over the years we've been doing this is that any company which doesn't test is less likely to get me as an employee. Like their answers to questions like what they use for source control etc., whether a company tests tells me a lot about how good they are at the business of software development.

So, what should you do? You should do what works for your organization, but my advice is to continue the tests, but do them onsite, as part of the interview. Tell them in advance that that's part of the process, and do it happen before they meet with upper management (but you should meet them first to help set them at ease). And really: skip the interview with upper management if they fail. Don't worry about popularity with candidates or posters on the internet. The cost of a bad hire is way worse than the cost of delaying until you make a good hire.

Answer 5

Why no at home tests?

Even if we ignore the possibility of cheating, and the reverse-filter which causes good and honest candidates to avoid companies with at-home coding tests, the value of at home coding tests is limited.

If it's for a senior position, a senior developer with people skills will be able to tell within less than 10 minutes if the senior developer on the other end of the phone is any good, or is just making up stuff. We won't know how good exactly, but we'll know as much if not more than every conceivable at-home coding test tells us.

If it's for a junior position, we don't expect much in terms of technical expertise. We're looking for enthusiasm, willingness to learn, and talent - none of which are spotted from an at home coding test. There are too many things juniors are allowed to be oblivious to. If we hire them we need to train them anyway.

How to test instead?

As a filter, just give them 10 minutes to solve a FizzBuzz variation on site during the first round of interviews or, if you get swamped with good CVs and need another filter, do it over Skype before the first real round of interviews.

Once they've passed the filters, you need to know more about the coding abilities of your candidates. I recommend spending 30 minutes to at most 2 hours of either pair programming or code reviews - actual real work, rather than an exercise. Repeat 1-2 times with different partners. The advantage of pair programming and code reviews is that the developer already has sufficient knowledge to contribute.

Don't worry about the test being different for each hire - the goal of the hiring process isn't to find the person who scores best in a couple of measurable and repeatable tests. The goal is to hire a single person who will do the job well.

Answer 6

Here is another point that I don't see covered in the existing answers (which I think cover the topic well in general). I would take a close and serious look at how long it actually takes people to complete. I have had four job interviews during which I had a programming exercise to complete, each of which was done differently (and each had its own advantages and disadvantages). Two of the four (numbers 3 and 4 below) took much longer than they said it should, and both of which I ended up giving up on because of the difficultly level involved. I've described them below and ranked them from best to worst from my perspective.

1. During the technical interview, they sat me down at a computer that had a pared down version of their code base and had me solve three relatively short problems related to it (find and fix a bug that they had purposefully added, add a new field to an info table, etc.). They gave me exactly an hour to do it, and after an hour they went over my solution as well as how I approached the problems. This gave them more insight into me as a programmer, while respecting my time by keeping it short and to the point.
2. During the technical interview they had me work through a problem they had encountered during development on a whiteboard while being able to bounce ideas off of them. This was the shortest option, while still giving them a chance to see how I work through problems and how much I can actually do the necessary job.
3. During the technical interview (for a junior Ruby on Rails Web Development position) they tasked me with building a web application from scratch that would navigate to a website, fill out multiple forms as they were presented, scrape data from the resulting page, and present that to the user. They said this should be a quick exercise, and it may have been for a senior level web developer, but having had only a year of professional experience at that point, I spent four hours trying to get it all to work before giving up and going home (all of the interviewers had left hours before I did because it was an afternoon interview, they said I should just save the finished program when I was done). This is a ridiculous assignment for the listed position, gave them no idea of my coding abilities, and seemed to me like they were just trying to get free work from the deal. Needless to say I didn't even want the job by the time I left that day.
4. Before even having a technical interview, they gave me an assignment to create a web application that would take advantage of an API their company used to "do something interesting". This was exactly as broad as it sounds. This required me to do the following before even attempting to create the application: create a developer account for the API, download the API development kit, create a publicly accessible web application (with another developer account), learn the API, and create a data repository to access with the API. This of course took many hours just to get started, and shortly after getting started on the web application itself, I got a different job interview and shortly thereafter a job offer, so I discontinued work on the assignment. This is a crazy thing to have as part of an interview process because who wants to put all of that time and effort into developing a program just to have a small chance of moving on in the interview process?

So to more directly answer your question, should you have a programming exercise? Yes, but make sure it is tailored to test what you actually care about, and not require a ton of extraneous work for the interviewee. Do you want to know about their algorithmic thinking? Give them an algorithm problem. Do you want to know about their coding style? Give them a coding problem. Do you want to know about their development process? Discuss their process with them as they work through a problem.

Answer 7

Let me start off with:

• I've been given tests to complete at home that ranged from 15 minutes to 10 hours.
• I've been given online tests to run through.
• I've been tasked with writing out the answer to FizzBuzz and other fashionable internet tests on whiteboards.
• I've been asked about the square vs round manhole covers.

In short, I've dealt with pretty much all the different ways developers like to handle interviews. To be quite honest - I seriously doubt the vast majority of people interviewing me even understood what the potential outcomes of each of those tests were and ultimately just hired people on whether or not they "liked" them.

Before you even put up a job listing you should sit down and go through exactly what it is you are trying to hire and "developer" isn't an answer, at least, it's not a proper one. A good answer to this would be something like "mortgage domain expert".

Finding someone that can write a bit of code or search google for how to apply a particular pattern is trivial compared with finding someone that's been in the business you are in and can leverage that knowledge. In other words, if I was hiring for a mortgage documentation company, I'd take someone that could talk about the difference between 15 year fixed and and ARM loan over someone that could write a bubble sort algorithm in 2 minutes. The reason is that "normal" business people make all sorts of odd demands and domain experts can more easily get to the heart of what's needed whereas someone that knows nothing will happily add things that are useless or make the app really bad.

Getting back to the questions, only ask go / no go questions.

Is it critical that the person can tell you the difference between a virtual and abstract method? Might be, usually isn't. I'd wager a good portion of developers barely even know when to use those keywords but they aren't your superstars, they are the rank and file who use can't code without the visual designers.

Is it critical to spot when a query is open for sql injection? For a Sr position - absolutely; for a Jr. position no. This is something that is easily trainable and handled via code review. Hence the reason you want the sr. to know it inside and out - so they can train the juniors.

Is it critical that they know the exact maximum value of an Int32 data type? Not normally - that level of trivial knowledge is what google is for; but maybe your requirement is for coding on memory constrained devices - in that case: absolutely.

I interview and hire developers. I don't send home tests - it's too easy to have a friend help. I don't use online testing sites - given how the scoring has to be automated it's trivial to game. I don't ask candidates to write out the answer to fizzbuzz - just about everyone has seen that test and should know the answer by heart; everyone else entered the field in the last year and are jr.'s anyway. I also don't ask trivia questions - being able to recite the answer usually just means you've heard the question before.

What I do to interview people:

• I ask them to describe a prior project or two. All I care about at this point is to make them comfortable and to get them talking. This is a go/no go because if I can't understand what they are saying then I can't work with them.

• I ask them a few specific questions in the tech I need them to use. If it's SQL server, I'll ask about updating on a join. If it's HTML, I'll hand them a 10 line html file with a couple css classes and ask them what the output is. These are trivial questions to people with experience in those areas and quickly weeds out pretenders.

• If I'm looking for a Sr. dev then I'll ask domain knowledge questions. Not edge case things but rather core principals. If I need you to lead a project on an accounting back end then you better have a grasp of basic accounting principals.

• If I'm looking for a Jr. dev then I'll ask about pet projects. all I want to know is the type of tech used. This should clue you into what they really want to do. In other words a C# developer isn't likely to be doing pet projects in php. I honestly don't expect too much from a jr dev except to be trainable. If they are actively working on a pet project then I can train them. If they are the type that needs people to tell them what to do there are far larger companies these guys can work at.

I don't make these questions up on the fly, the potential answers are considered ahead of time and fit a go/no go pattern. If a question doesn't fit that then it's not relevant. I also ask every candidate the same ones unless I'm 100% convinced I'm not hiring them at which point I'll stop the interview.

If for some reason you still insist on sending home a test - make sure that the skills required to successfully complete that test in a reasonable amount of time are actually skills you care about.

I had one company give me a take home test that involved writing a custom cryptographic service provider. I completed the test because it was somewhat interesting; they hired me. At no point in my time there did I do anything even remotely related to security, cryptography or even math based beyond adding a few numbers. I wonder how many people they drove off with that test?

Answer 8

Coming from the standpoint of a person seeking a job, I generally avoid places that take over 1 hour to code. One time I went to this place that required a nearly 3 days worth java coding project. I did it all and the guy was even impressed with it but they told me they hired someone else after the second interview stage. So after that, if I come to a company I would ignore/pass any project that requires more than a couple of hours to complete. My time is just as valuable as theirs and I rather not waste it on things that won't get me anywhere.

With that said, if your coding exercise is too long, maybe people aren't willing to put the time in. I would attempt to reduce it such that it takes an hour at most but at the same time demonstrates an understanding of coding and maybe putting a deadline on it like, "Please complete by COB tomorrow" or something. They can still "copy-and-paste" something online but make it hard by being specific rather than something you read online.

Answer 9

I used to be a firm believer in coding tests and whiteboard coding, but I've started to realize that they're pretty much useless, because

What are you testing, anyway?

A whiteboard test, or short programming test gives you some insight into the individual, but really not that much. Unless you plan on having someone spend their time writing code on a whiteboard or FizzBuzz style code.

What do you want?

You want someone who is:

• passionate
• willing to learn
• able to solve problems*
• resonably technical
• going to help your team improve

*_{Note, most developers solve their problems by knowing what terms to search for in Google.}

The last thing that you want to hire is someone who is not a good fit for your team because they will drag it down.

How Can You Test For These?

• Ask them about a project that they enjoyed. If they seem reluctant to talk about it, try expressing gentle disbelief, e.g. "You can't do X, can you?" If it's something they're passionate about, they will correct you. This will also help you learn if they're technical or not.
• Ask them about things that they've learned recently. Or what they learned from the project they worked enjoyed.
• Ask them about the last time (or a time) where they were missing some knowledge. How did they get the information they needed? Did they go to a teammate? Did they search the internet?
• Ask them if there's anything they would want to see improved on their current/last team. Did they need better commit messages? More code reviews? More testing? More automation?
• Ask them what technology excites them, and why.

If you have someone who is comptent technically in on this conversation it will be the easiest thing in the world to tell if this person will be the worst. An example: we had a kid come in and interview - he said that on a scale of 1-10, his Java skills were like a 7-8. I don't think he even knew that Java was compiled to a jar file which was later compiled to machine language by the JVM. I'd rank myself maybe a 2 or a 3 and I know that.

He was actually asked the same question by our CTO in an interview the previous day, and our CTO called him on it and explained that there was no way he was a 7-8.

Our CTO also asked him about his favorite project, which had to do with hand-held scanners. But he couldn't explain anything about how they worked, or the fact that they used polling to prevent busy-waits. He couldn't explain a single technical thing.

That guy didn't get hired.

Figure out the kind of attributes you're looking for, and then figure out how to test for those

But I really need to see how she codes!

Okay, that's fine - but unless she's going to be coding in a silo, the best thing to do will be hire her as a contractor for a small, well-defined project. Maybe you're adding the capability to download some files from an FTP and then dump them in your database. Something simple, that doesn't require much/any domain knowledge.

Have your candidates work with an existing employee or two, and pay them for their time. You'll get to see exactly how they work, and how well they work with the team. Do they communicate well? Do they get frustrated easily? Are they persistent?

There are things that you can do to hire better employees... but a programming competition probably isn't one of them.

Answer 10

As others have noted, some developers may be put off by a 1-2 hour programming exercise to apply for a job. What may work better is having a white-boarding session, where the candidate solves a problem on a white board during the interview. This allows you the opportunity to have an in-person interview while also making sure they have technical chops.

These problems should not be extremely difficult, or designed to trip up a developer. A classic example is FizzBuzz. This allows you to see how they think, and also know that they can at least program and don't need to google how to do a loop.

Answer 11

DO NOT GET RID OF THE CODING TEST!!!!

If you want to be a programmer you have to write code that does something. Answering trivia questions is not that important. Chatting about past experiences is good, but it is in no way a substitute for being able to code.

Do what it takes to attract people to want to take the test and get this job. Make it shorter or pay them. Bring them in on a weekend so everyone has time to observe this person in action.

This is really contingent on having people who can interpret the results of the tests/observations. You may want to hire a third party to do this screening if you don't know how to do it. Just getting the answer right, is only half the battle. Developers need Googling skills, but there is much more to it. If one programmer can do the test faster, that is a plus.

Depending on your project, you may prefer fluency in a language, so the dev can start working efficiently right away. On the other hand, you could suffer in the long run when you opted for the less skilled programmer who is going to get stuck with a key feature of your app or they build such a poor structure, it's difficult to alter and maintain. You may have been better off with a good programmer who is willing to adapt and learn the language needed. There are many devs who will absolutely run away from certain stacks, frameworks, languages, tools and operating systems.

Developers are critical positions that are very expensive to fill. I understand your concern over chasing too many good ones away, but you only need one. Don't skimp.

Answer 12

In my opinion, the problem you have here is not necessarily the programming test. First you have the telephone technical interview and then a work from home test before a face to face interview. It sounds like you are keeping your candidates at a distance and leaving it to the last minute before meeting them. At what point do you expect candidates to decide that they want to work for you ?

I assume your recruitment advertisement is similar to most and so focuses on location, salary and a (wish) list of skills. Candidates don't really know what they would be working on, anything about the environment or people they would have to interact with. You haven't sold the job to them yet here you are asking them to prove their technical capabilities twice before they get to ask a single question about the work.

I suggest you try changing the format of the technical phone interview to be a 30-45 minute chat about the job including plenty of opportunities for questions from the candidates, then 15 minutes of technical questions as a screen so you still get a chance to pick the better applicants without making it too onerous.

I would also consider moving the programming challenge to be done onsite prior to interviews. It would seem more achieveable to the candidates, give them an incentive to stay on track with the process and you would get the benefit of observing the true time spent on the challenge (I think you might be surprised).

Answer 13

I don't like take-home tests as part of interviews, for many of the reasons people have already mentioned - extends the hiring process, devalues applicant's time, may not get a call back anyway, etc.

My main issue is that it's unrealistic to how the team actually works and makes the interview process one-sided. An applicant wants if this place is a good fit for them, including the culture, how the team approaches and solves problems. You primarily are also looking for fit, including how they work and if they have the right skill set. A take home test doesn't provide an applicant an opportunity to evaluate the soft qualities of a workplace, and employers don't get to learn how the applicant approached the problem.

A better solution might be to give the applicant a more open-ended problem that can be solved in any type of creative way. You could even restrict it to X language. Rather than email it in, invite them back to present it to yourself and upper management. It gives them autonomy and incentive to do well, because it promises another interview and shows you care to know what they think.

If you must use a test to screen which candidates make it to interview with upper management, I would include the test in the interview so you can discuss their thought process together.

Answer 14

Do you want to hire programmers who can't program?

I'm going to venture that you do not.

Hiring programmers who can't actually solve problems and write code is a good way to ruin a tech company. And you're not going to be effective at weeding out the programmers who can't actually program (and there are a lot of those out there) if your hiring process doesn't include some sort of programming test.

Are you willing to lower your standards because everyone's trying to hire programmers?

Maybe you are, but I don't think you should be. As has been noted in the comments and answers, there are candidates out there who aren't going to bother doing programming exercises as part of an interview process because they just don't need to in order to get a job.

But are those really the people you want to hire anyways? The ones who follow the path of least resistance, do whatever is most beneficial to them in the short term, and don't really care enough about your company to complete a simple programming exercise? Those don't seem like positive attributes, and they don't provide much confidence in terms of being able to retain those candidates long-term (which is also important for a tech company, as learning curves tend to be steep and the cost of replacing existing staff is very high).

So let those other businesses have the programmers who can't even be bothered. You don't want to hire them anyways. Unlike them, you have a plan. One that's not based on the "a programmer is a programmer" fallacy. Your focus should be on quality and sustainability, not bodycount.

Is scaring off candidates a problem?

Generally no, as long as they're being scared off for a good reason. You don't want to hire people who aren't up to scratch. And some of the people who say they "can't be bothered" due to high demand might actually be using that as an excuse to cover up a "not really that good at programming so would need all week to complete a 1 hour exercise" situation.

It's good to scare away those candidates. You want to hire the capable, motivated candidates. As long as you're not scaring those ones away too, you're good.

Every candidate you don't scare off is one you have to try and evaluate. And that can be hard to do if you're not giving your technical candidates any technical exercises to use for evaluation.

How can I improve our hiring process?

Check the content of your programming exercise. Is it reasonable and appropriate for an interviewing context?

You don't want something that's going to take days (or even hours) to complete. What you want is something simple to weed out the people who just can't program, ideally with enough room for nuance that the people who can program really well can differentiate themselves. Keep in mind what you're trying to accomplish (weeding out unskilled and non-serious candidates), and ensure your content is tailored to that goal. Don't go overboard.

If you've already got some technical staff, you can use them to sanity-check (and/or help design) your exercise.

And also consider how you administer the exercise. If you just give them some documentation and say "here, do this over the next week and e-mail it to me", that's probably only going to be minimally effective.

Better might be if you can run the exercise through a web portal, which candidates can check in and start the exercise, and once they start a timer starts counting down from 1 hour. Then they either submit something within that hour, or not. That's less open-ended, better retains the candidate's focus, and provides a clear deadline/timebox both so that 1) you're not left waiting around all week for a result that's never going to come, and 2) unqualified candidates don't throw away a week of their time trying to complete your programming exercise. They get 1 hour, they either solve the problem or they do not, and you know the outcome immediately.

And even better would be to bring them in for an onsite interview. Introduce them to a member of your development team. Shut them in a room together with a workstation. Have your developer start with some general/soft interview questions, and then they can pair-program with the candidate to solve the programming exercise. This will tell you not just whether or not the candidate can code, but also how well they work with your team. Your developer should also be able to glean a lot of additional information that you just won't get by looking at a bunch of code that a candidate wrote and then e-mailed to you.

Bottom line

No, you don't want to get rid of your programming exercise. But you may want to review it for appropriate content, make sure it doesn't take too long to solve, and also look at how you're fitting it into your overarching interview process.

A self-directed take-home exercise probably isn't the best approach. But the solution to that isn't to scrap the exercise entirely. Not unless you're okay with hiring crap programmers, at any rate.

Better to scare away a lot of bad candidates and a handful of good than to open up the floodgates and hire a few bad ones.

Answer 15

As a direct reply to Bobo's answer (which is the accepted answer because the guy wrote it and accepted it himself, which frankly I feel is a bit pathetic):

You are coming from a totally wrong premise. I don't want to work for you. Where do you get that idea from that someone wants to work for you? You are just one of many companies offering a job. I don't want to work for you. I want to evaluate your company, and if you come out on top of all the others, that's when I want to work for you.

There's a dozen companies where I can work, you are just somewhere in the queue. I first look at what companies there are, send them my CV, they can read it and be suitably impressed or not, then you usually have a quick phone conversation where they show me that they have an interesting job and I show I have the skills, and any coding test might come at the very end.

Your take home test puts you at the very very end of the queue. To compensate you'd have to post a salary range that is £10k higher than others. Finding a job is time consuming anyway; someone who makes it ten times more time consuming is at the end of the list. If I have the choice between sending a CV and having a phone interview with ten companies, and doing homework for you, guess what I'll do.

So what happens is you find candidates that want to work for you because they won't get a job elsewhere.

Answer 16

One thing no-one appears to have mentioned; even if the test is not your issue, you should be looking for other ways of attracting talent.

If you are hunting for good people based on their existing published work, you don't need to run the test on them.

If you're just being sent people via recruiters and filtering, you are vulnerable to the fact that your needs and the recruiters' needs do not perfectly synergize. They want to place someone with you. You want a top level engineer. If they can find a top level engineer that's awesome because you'll come back to them, but if they can't (and top level engineers tend to have stuff going on that leaves little time for interviewing), they'll settle for just placing a moderate engineer with a nice suit. The loss to them is a bit of long-term rep, but that's minor compared to missing their targets. If this is provably not true in your instance, hang on to that recruiter and never let them go(*), recruiters who are prioritizing a long-term relationship over targets are precious diamonds in an ocean of coaldust.

What you want to be doing is finding provably interesting candidates. Hunt through StackOverflow and GitHub for top engineers (I hear StackOverflow has a tool to help with this), check for technically interesting companies that produced good software but who screwed up their financials or monetization, and just laid off 10 top level engineers. Spend time working at universities assisting with final year projects. Identify good potential candidates and befriend them, preferably in person, alternatively via remote; even if they have offers, good engineers hang out with good engineers. Also, they can tell you how they feel about your hiring process.

Does this sound like a lot of work? It should. One of the reasons that hiring seems so "hard" is because you're trying to do it as efficiently as possible. The more time, brainpower and resources you divert to it, the easier it is. Whether those resources would be better spent on shipping product is the eternal management question. But if you're spending a lot of time on "crap programmer filtration", that's burning money. At least the steps outlined above have inherent value beyond the hiring process.

(*): Hell, hire them.

Answer 17

I think that you have phrased your question a little bit wrong, but the way you've phrased it reflects a common misconception about hiring programmers. Are the candidates being "scared off" by the programming task, or are they filtering your company out of their own consideration because of the task?

An anecdote to demonstrate my point: while job hunting not long ago, I saw a position for a company which seemed average. The way they described their programming process made it sound quite good, but there were very few details, so I was skeptical. Maybe they were a good place to work, maybe not. But I figured I'd see about getting to a phone screen so I could figure out the details and see if they were as good as they seemed.

I clicked on the job posting, and was immediately asked to write a cover letter. Ugh. I think every candidate hates cover letters. I didn't know this company or use their products, so what could I say about them? I googled them, read through their website and product offerings, figured out where I was likely to fit into their org chart if hired, and came up with a few paragraphs "selling" myself.

Next I provided my resume and access to my LinkedIn - but immediately afterward they asked me to fill out my relevant work experience with dates and descriptions. This info is both on my LinkedIn and my resume, it was ridiculous to have to provide the same information 3 times. I closed the browser tab. 5 minutes later I was applying to another company that offered some really cool benefits which the first did not. I could actually apply to another company with better benefits faster than I could jump through the hoops the first company wanted from me.

You need to be sure that your candidates are invested in your company in particular before presenting them with any hoops to jump through, or else they won't jump. Do you do this?

Examples of quality benefits I commonly see tech companies offering:

1. Remote working.
2. Free computers/monitors as a signing bonus.
3. Company contributes to respected open-source projects.
4. Reimbursement for professional training and/or conferences.
5. Catered lunches.
6. Flexible hours.
7. Opportunity to work with new or unfamiliar technologies.
8. "Startup culture" - aka lack of politics/bureaucracy.
9. Company Equity.
10. Name recognition: your company or your product is well-known. Candidates like to mention where they work and hear people respond with "Oh, neat! I like their products."
11. Charitable or revolutionary company goals/vision. People like to write code that makes people's lives better.
12. Above-average pay. Money covers a multitude of organizational sins.
13. Annual company retreats to cool places.

This list isn't comprehensive, but if your company doesn't offer one or more things like the items on that list, then any obstacle in your hiring process could send a candidate looking for one that does.

So let's say that, for whatever reason, you don't or can't offer incentives like the above for your candidates to convince them to spend writing code for you for free. Then what can you do? I have two alternatives most candidates will prefer over busywork programming tasks:

Alternative #1 - Pay them an hourly rate to do your programming task as though they were a contractor. This encourages them to take it seriously both for professional reasons and because... they're getting paid. This costs you money, but so does any form of recruiting. If you're really good you can even find a way for them to diagnose and fix an actual bug in your code, in which case you're getting something useful for your money.

Alternative #2 - They have probably already written code for free which they will show you if you just ask. Most programmers have code on Github, Bitbucket, Q&A websites like Stack Overflow, or could provide you with some code they have not already published.

Why make them write code they don't care about when you could let them share a passion project with you instead? It's guaranteed to be less boring than reading yet another solution to the same generic problem for the 100th time. And since the code is already written it saves both you and your candidate time. Plus you'll get a look at what sort of code they enjoy writing, which gives insight into their personality and how well they will fit with your company culture.

Answer 18

What is your business problem?

Disregarding all the arguments for or against particular tests, it all comes down to a tradeoff - more filters will scare off some good candidates, less filters will let through poor candidates that you might have to replace soon after hiring.

It all comes down to looking at your business situation instead of general practices.

Do you currently have a problem where you lack qualified candidates and are unable to hire as quickly as you need to meet your business goals? You need to ditch this initial programming task.

Do you currently have a problem where you are not satisfied with the quality of your recent hires? Then you need to implement more filters like this.

Do you have both of these problems, and both are equally painful? Congratulations, you have found the right balance for this tradeoff.

Answer 19

I really think it can help establish who really knows the languages listed on their resume.

If that's really your goal, I would consider a different approach. As other answers indicated, YES, you should always have a coding question, but coding questions rarely get into details of the language. Some questions I've seen that are useful for testing this:

1. Compare and contrast Java and C (or whatever two languages are relevant, are on the candidate's resume, etc.)
2. What features do you think should be added to the language? (Or better yet, what do you think about this particular proposed/recent change to the language?)

Engineers who've seen a thing or two know how to answer these questions pretty easily, and in only a few minutes.

Answer 20

IMHO there is almost 0% chance that a fresh college graduate would be able to do a tough programming code at an entry level. If your coding test takes one week time to complete then you should clearly mention in your requirement that you people are looking for programmers with at least 2+ years of experience because I think that much experience should be required for completing a code work which requires one week to complete. And I think if you are looking for fresh graduates then design your test accordingly and you can find lot of idea on the internet or else you can ask senior programmers working for you to design a suitable test for fresh graduates.

Answer 21

Thought that I will answer this question, it has been a year since it has been posted, and we have stuck to it.

FINDINGS

Positives of approach

1) Take home test weeds out the demotivated candidates, and replaces them with candidates that really want to work for you. This alone makes it a worth while thing to do since motivated people = productive employees. If they cannot be bothered to do a 1 hour assignment then that says a lot about their attitude to getting the job.

2) I agree with others, that the take home test should not be longer than an hour - mine is very simple. I have had the following results from adding it into the recruitment process-

a) Some candidates do not complete it. Not worth hiring.

b) Some candidates attempt but complete it poorly. Not worth hiring.

c) Some candidates cheat, at which point it is worth asking follow up questions about their assignment. We did this with a recent candidate who then did not bother to respond to our email about the assignment. Not worth hiring.

d) Some candidates upon hearing there is a technical assignment suddenly withdraw their application, where previously they were showing A LOT of interest. Probably dodged a bullet.

e) Some candidates do extremely well, comment their code and in one or two occasions provide documentation. Worth hiring.

Negatives of approach

1) Application drop outs from candidates being put off by the take home task makes it longer to find someone suitable. BUT on the flip side positive for the company since it reduces the probability of a bad hire which is dangerous.

2) Can't always tell if someone has cheated, but that's why it is often backed up with a technical phone interview.

Outcome of this approach

One of our employees who I hired using this approach has turned out to be a star employee. He is still working for us after over a year. He is reliable and technically talented.

Answer 22

Unfamiliar mouse or unexpected keyboard layout (especially Mac vs PC), or different IDE may slow performance dramatically without any difference in competence. Also, a complete application often requires lots of boilerplate code that a developer may not have enough time to type in or even not remember. Starting a new app completely from scratch is actually a very rare task; most of the work concentrates on extending or improving the existing code.

Hence I suggest to give only very short and more carefully prepared tasks during interview. Best is to ask writing a function that must take given parameters and return the explained result and I would advice to do on paper, avoiding computer at all.

Answer 23

I'd send them to an online quiz where you can filter out people who have no clue. At least you'd have a sense whether they know what they're talking about.

Early in my career headhunters told me "you're up against people who read the box and put an application on their resume." I assume this may still happen to the young and naive, but once you get trashed in a few interviews you learn this is bad advice...

Answer 24

I recently got a take home test. It was full blown application that needed to connect to a socket server that had to simulate a slow feed. Client had up update dynamically, be able to cancel the feed, and write and read XML.

I kind of want to learn sockets anyway as I am thinking about using them for a poker server I am writing.

I wanted to learn XMLreader and XMLwriter.

At first I thought forget it. But then I saw it as a chance to prove what I can do. I don't have a CS degree so I miss some theoretical stuff. They asked what are the 3 pillars of OOP and wanted to say who cares.

My message is people that want the job should complete the test.