How do I determine if a programming task is doable during an interview?

Question

We are having interviews for a new developer position. During the interview, I give the candidates, who tell us they already have experience in .NET and SQL, certain programming tasks, which I think should show the most basic of proficiencies if completed in about an hour.

Since I am the only developer right now, no one in the company is able to tell me whether the tasks are suitable for an interview or not.

During the tasks, I am available for the candidates' questions. I tell the candidates that since the questions are dependent on each other, they should ask if anything is unclear, and I help them if they are stuck, expecting them to be able to at least complete the next step. I also tell them that if they have problems with the syntax or with typing on an unfamiliar keyboard, they can instead discuss with me how they would implement it.

For some reason, all our candidates failed to complete more than half of the steps. How do I determine if the tasks are too hard, or if we just got unlucky with our candidates?

Answer 1

The coding part of the test took me 5 minutes, it would have taken maybe 10-15 if I had typed out all the pointless properties and assignments and implemented the method (that's never called). Double that for a potential crappy environment and interview nervousness and the time is fine. The SQL one should be just as easy.

I think you could make this significantly easier by omitting all the pointless properties the person needs to type out. If the person can implement and assign one property, don't make her type out the code for all of them.

Apparently, it works as a form of FizzBuzz. It weeds out people without a basic understanding of the language. That's a good thing. Just imagine you had hired one of them!

The fact that people fail your test shows that your process up to the interview is flawed.

You should ask yourself how it comes that you get to interview people that fail such an easy test. Did they lie on their CV? Is this common in your region? Is there a way to find out before? Maybe you can find a few simple questions you can ask on the phone? Or did you invite the wrong people? Did they actually say they knew C#, or did you assume that? I'm afraid this is where I cannot help you, because where I live, people generally do not lie or exaggerate in this way on their CVs. If they say they can code in C#, they can.

Please note that once you get to the stage where candidates pass this very basic test, you may want to refine it. Being able to do the basics is cool, but maybe not enough for your company.

---

As to the edit of the question how to find out if something is good as a test, just do what you did: ask your peers. Whenever there is a test to be prepared, I always grab members of other teams and try it with them. Have a few people do it and include the feedback they give on time and difficulty. As you are the only developer, you will need another way to find peers to do this: maybe a user group, your old classmates or a forum.

Do it as you would deliver software: Gather requirements (what do I want to test the candidates for), plan a test (prototype), let testers do it and gather their feedback, then refine the final product as many times as you see fit.

Answer 2

I'm going to answer this question based on the information before the edit, as I personally feel it was still vital to the answer.

Have you heard of Pseudocode?

When I got hired for a software job I only managed to get half the tasks working. Mainly because I had to code in a language I never saw before and the syntax was different (I told them beforehand and they told that wouldn't be a problem). So for every piece of code that I couldn't get working I wrote some pseudocode.

example:

// Place a for loop here, For as long as i < [var].Length then do this other thing. 

The most important thing is that you get inside the software engineer's mind. Know how he intends to code. After that you need to ask yourself how easy it would be to bring him to a level where he can be placed to work on larger projects. Usually this will take between 1~2 months anyway.

Furthermore, I would HIGHLY recommend to let him use the internet for code referencing. Coding requires a HUGE amount of looking up codes in order to learn them.

Once he's done, and you're worried he just copied some code he doesn't understand, just ask him about the code itself. Why did he use this variable here? Why did he name it like that instead of fully spelling it out? Why use a for loop instead of a while? etc etc. These might be silly questions, but if he's bullshitting you, he shouldn't be able to answer any of this.

Also, try to make your steps as independent as possible and then leave him to his thing while you grab some coffee and do other stuff. If he gets stuck on anything, he can simply skip it and proceed with the next. A lot of coders don't perform well if they are being watched. So by enabling him to do his own thing, he should have that chance. I'd still pop in every 15~30 mins in case he has any problems/questions. Also, this allows you to see if he's a coder that will keep asking questions before trying to find answers online first.

As for the SQL tasks, I feel like you were too easy on them. I'd show them a small SQL database model and tell them you want to add a certain item, but don't wish to edit any of the current tables. Asking him what his suggestion would be in order to implement the change. And also why. basically talk the information out of him. I'd still ask how to make an update/insert/create query though. He can find that stuff within 2 seconds, but if he can't it's an easy way to see if he's bullshitting you.

Bottom line is, you need to understand how he thinks. Not how good he is at coding.

Also, you may want to tell him before the test, what kind of things he can expect (not in detail, just the outlines). That way he may prepare himself better.

Answer 3

I think you have gotten a lot of great feedback related to the specific questions you asked the other candidates, but I don't see any real answers that address your stated question: How do I tell if a set of interview questions are too hard?

One obvious way you can answer this question is by soliciting feedback from other developers. You said that you are the only developer in your organization but that doesn't mean you don't know or can't meet other developers. You can ask for feedback from friends who don't work at your organization. If you don't know anyone else that slings code for a living then you could go to a meet up and try talking to someone there about the task. Finally you could consider running these questions past a recruiter. Of course there may be a conflict of interest there if that recruiter starts sending you applicants who perform amazingly well on your test but don't appear to know anything else...

Another direction you could go is search for interview questions online. This does two different things. First it might give you ideas about the style of questions other people are asking. Secondly you can compare the relative level of difficulty between common questions and the questions you're asking. You might think that this route might encourage cheaters since these questions are "known" but anecdotal evidence seems to show that even though FizzBuzz is incredibly well known a significant number of people still fail when asked.

Finally, I would argue that "too hard" is really relative to the kind of person you want to hire. Don't dumb down your questions because you feel bad that people aren't passing. Instead set your questions at a reasonable bar to screen out people who aren't going to be successful in the position. I'm certain that a position at Google where the applicant will be designing advanced search algorithms isn't going to have simple questions like "What is one difference between a POST and GET request?" Instead I'm sure they are asked advanced algorithm and machine learning questions. Having questions that are too hard for people who aren't going to be successful, but demonstrate an understanding of the principles they will need to be successful is ideal.

So my advice is to not worry too much if the questions are too hard. Instead worry about what are the basic principles you are trying to screen for that a candidate needs to be successful. Once you've isolated that the only real hurdle is making sure your questions are clear. That being said in my opinion there is real value to finding out if a person will ask for clarification before building a grand solution that solves the wrong problem.

Answer 4

What do you want this development task to achieve?

You have a small window in an interview, to judge whether someone has fed you a load of rubbish on their interview.

If someone was able to complete your task, would you then use that to think they are technically capable for the job?

The stuff you are asking seems to me fairly pointless to be in a technical test. You are getting them to create a class, a method, write a couple of lines of code and then do some basic SQL tasks. It isn't really showing knowledge in my opinion.

A better approach may be to show them code and ask how they would improve it.

For example:

I have a c# array of objects and I am doing a foreach loop.

You are then waiting for the candidate to say: "Oh, convert that to a list of objects and you can then use linq to search the list.

You can show them a SQL Statement with a clear inefficiency and then have an idea of an answer to be: "Oh, add an index on the table, or change that where clause to whatever.

You could even have an incorrect join in the SQL and see if they pick it up, or something like that.

They may also give you a completely different answer that does/doesn't work and opens up conversation to other areas where they can demonstrate knowledge.

Obviously tailor the questions to work around that example, but the above, I think, has the opportunity to provide further insight into how someone thinks and prove they know basic code etc.

Answer 5

I have two comments:

1) DateTime's SUCK to work with, if you want to see how people might handle converting from one data type to another or how they would deal with creating an implementing a class then I would suggest finding another example to work with. Additionally, what you're describing may be more a case of something infrequently done than difficult. In situations like that a decent developer should be able to resolve the problem within minutes using Google.

2) Creating and extending a table/schema are rudimentary and anyone that's ever written a SQL query should be able to create one to return records like you're asking, so if people are failing at this then either it's a syntax issue (used to having a reference) or they haven't worked with SQL at all.

One suggestion I'd make is that instead of giving them specifics to present them with the scenario and see how they choose to handle it. For example, give them the task to design table/s for a company that include employees, managers, and a way to query for the organizational structure.

Personally, I would find it more useful to know whether a candidate would jam all the info into one table or break it down (and why). Whether they can write a basic query or create a table isn't very useful information, how they approach design is much more important IMO.

Bonus: don't forget that some people REALLY struggle in an interview setting. I have known people who were very comfortable interviewing and answering behavioral questions but would completely freeze/blank on technical things because they weren't expecting it or prepared for something else.

IT is such a large field that you can spend weeks studying all the most common quiz questions and coding problems only to be given the one thing you overlooked... that does awful things to people in an already high pressure situation.

So keep in mind that part of your job as someone interviewing is to be able to figure out based on the way they approach questions whether it's nerves or a lack of knowledge causing someone to have a hard time.

Answer 6

How do I determine if the tasks are too hard, or are we were just unlucky with the candidates?

1. How long is your interview? (For the sake of discussion, let's assume it's two hours long.)
2. How many programming tasks are you intending to quiz the interviewee on? (Let's assume it's four tasks).
3. Given four tasks for a two hour interview, could you complete each task in 25 minutes? Or if some are more difficult than others, could you complete all of them within 100 minutes? [Assume that you are seeing these questions for the first time.] If the answer is 'No', then the interview is too difficult. If the answer is 'Yes', then the interview is not too difficult.
4. Alternatively, you could ask a former colleague or programming buddy to play the role of a prospective interviewee and see if they can complete your technical interview in a reasonable duration of time.

Answer 7

One statistic is: how many fail? If you have had more than 10 fail what is the chance you really got 10 bad candidates?

Are they getting the answers wrong or just not finishing? In a strange environment 1 hour is not very long. I would only expect you could ask for like 4 basic tasks. In an hour you are only going to be testing some basic mechanics.

Maybe the questions are not clear and maybe the questions are too many / complex for 1 hour.

Could you post some of your questions?