Programmers who are highly recommended, but cannot handle paper test programming question

Question

tl;dr: How to deal with programmers who are highly recommended, but cannot handle paper test programming question

I interviewed a candidate for a programming position. He is a recent graduate from a college in computer science degree.

He has an excellent recommendation from his supervisor while he was interning for a programming job, and he had a string of programming side projects under his belt during his university time. Not very difficult kind of projects, but definitely self-contained and usable websites/mobile apps. Based on the resume he looks like a stellar fit.

The only problem is that when I gave him a paper test-- basically he has to use pen and paper to answer some programming questions, he struggled and couldn't get a single one correct. My questions all are very basic programming questions-- somewhere along the level of fizzbuzz, matching an element in the array-- the kind of questions that can be easily answered by those who take one semester of entry level programming course.

And I already mentioned that I don't care at all about the syntax/language. Pseudo code is good enough for me

He requested to take the questions back for answers, which I agreed. The next day he sent me the fully corrected solutions. He did the assignments on an IDE and managed to get all of the answers correct.

This puzzles me-- if he is as good as his referral recommendation implies, why he can't handle the paper test? Is it possible that there are some people who can code well in front of a computer screen, but when comes to actually writing programming solutions down on a paper, they struggle?

Edit:

1. I don't care about the syntax, this was communicated clearly to him
2. The questions level only make use of the most elementary constructs like for, if... nothing at all about esoteric library calls. Anything that requires working knowledge in a certain framework is considered not-basic. So I am not asking the test taker to memorize any calls to http method.

Answer 1

Is it possible? Sure. People can get so accustomed to an IDE with syntax autocomplete and syntax highlighting that they hit a mental block if they try to write code on paper or on a whiteboard.

Is it possible that he had someone else do the work? Sure.

Why not ask him to come back and do a similar set of problems with a computer with whatever IDE you want set up. If he's able to produce quality code in an environment that is much closer to the actual workplace, great. In the future, you might offer candidates a choice between using an IDE or doing the questions on paper up front if people have a mental block with paper tests.

Answer 2

This link from the comments https://blog.codinghorror.com/why-cant-programmers-program/ posts an interesting problem, one that is frankly frustrating to write on paper.

Why?

Here is the requirements in text:

Write a program that prints the numbers from 1 to 100. But for multiples of three print "Fizz" instead of the number and for the multiples of five print "Buzz". For numbers which are multiples of both three and five print "FizzBuzz".

Here are the requirements as steps

1. Loop from 1 to 100 and print the number
2. Check for multiples of 3 and print "Fizz" instead of the number
3. Check for multiples of 5 and print "Buzz" instead of the number
4. check for multiples of 3 and also multiples of 5, print "FizzBuzz"

This is where it falls apart. Programmers don't think how to do all 4 steps at once, they do one step at a time. On a computer this is easy, write the first step, then edit in the second, then edit in the third, check if the 4th worked.

But on paper, you would have to write it 3+ times, either on paper or in your head. This takes a huge amount of time because this isn't how programmers have been trained to think. And any small mistake is another rewrite.

Programming is an art of making 100's of small mistakes quickly and fixing them. Duplicating this in writing takes a ridiculously long time. A senior developer taking 15 minutes is completely reasonable.

This is also lot of stress to pile on during regular interview stress, an activity that is typically already difficult for programmers. So they freeze up.

Answer 3

I wrote my first programs in 1967, and got a programming job in 1970. For the first few years of my career, I wrote my programs out on coding sheets.

There are three significant differences between then and now, any of which might be a problem for a programmer with only IDE experience:

1. Remembering outer structures. In an IDE, you can set up e.g. an if-then-else before filling in the content. On paper, you need to remember where you were with each incomplete structure.
2. Looking things up: IDEs offer choices and word completion that are not available on paper. When programming on paper I needed reference manuals within reach for anything I did not have completely memorized. If you must do paper programming tests, provide the appropriate references either on a computer or on dead trees.
3. Handwriting. I could produce legible block capitals very fast when I was doing that all day every day. When faced relatively recently with some academic exams that had to be done paper and pencil, I prepared by practicing writing, including writing mathematical notation. It was many years since I had done much handwriting.

Unless you really need the ability to program without an IDE, you should set up a programming environment for your tests.

Answer 4

if he is as good as his referral recommendation implies, why he can't handle the paper test?

Because there is no need to remeber everything you can google in no time. I find questions about algorithm complexity, finding array elements or sorting etc. intimidating and unprofessional. These are things you can read about very quickly if you need them.

Is it possible that there are some people who can code well in front of a computer screen, but when comes to actually writing programming solutions down on a paper, they struggle?

Yes, it is possible because it's much more important to have a general idea about what you are doing and being able to think abstract, take the long view or see the big-picture and be able to find the necessary information when needed than know by heart every algorithm. Pure encyclopedic knowledge is worthless when people don't know how to apply it properly.

Too many interviews focus on such things as whether you know how fast a certains sorting is (which you can find within few seconds) rather than to evaluate how people think.

Answer 5

if he is as good as his referral recommendation implies

That's because your expectation is too high for both the candidate and the supervisor. The recommendation letter, if unbiased merely mean the candidate's level is satisfactory relative to the person who wrote the letter. That doesn't mean the candidate is skilful relative to your standards.

For example, an 14-year old kid who can compile a hello-world program may impress his grandmother, but not you.

You probably have better technical knowledge than the supervisor, or at least someone who has higher expectation in excellence.

Answer 6

Is it possible that there are some people who can code well in front of a computer screen, but when comes to actually writing programming solutions down on a paper, they struggle?

All sorts of things are possible from mental block to cheating. But none of them can be known for sure and none apply to what you need to do which is gauge competence and suitability. All else being equal this candidates CV should be at the bottom of the pile.

Reccomendations are just paper, better than nothing but not to be relied upon fully. I've seen glowing recommendations given to people that the recommendee barely knows or doesn't know at all.

Answer 7

Paper test are not bad and pretty much standard across industry in the area I come from. My suggestion would be to ask the candidate to write algorithms rather than actual code. I can buy that the candidate does not know the exact syntax. But they should be able to provide algorithms/ pseudo code.

There could be two potential issues

1. Candidate may actually be cheating.

2. Candidate will take more time to complete the assignment that usual , which means he spends his time in reading post the problem is given.

It comes down to the way you have been groomed. In our days, lot of focus was on algorithms and we were taught to write algorithms on paper. Before IDE's came into picture, we had to write code in notepad which made our hold on syntax much stronger than someone who has learnt on IDE. Getting syntax right is not very important, but if the candidate is struggling to a point that he/she cannot answer a single question, in my opinion, then the candidate needs to work on basics of programming. Chances are that they will struggle and take a lot more time to complete their assignment that what would normally be expected.

Answer 8

"Is it possible that there are some people who can code well in front of a computer screen, but when comes to actually writing programming solutions down on a paper, they struggle?"

Absolutely possible. Different people work in different ways. Their brains work in different ways. And some people just have problems writing things down by hand.

I can type things without having to think about the typing process at all, it goes straight from my brain to the screen. Writing on paper is harder. (Speaking things is a lot harder, turning thoughts into speech takes a large percentage of your brain power).

So if you want to observe people, you observe them using the tools that they are used to, and there you see how well they can do. If someone is bad writing on paper and good using an IDE, hire them. The other way round, don't.

Answer 9

Anything the applicant could have done without you watching over him is of more or less zero relevance when it comes to gauging his performance. Anything you do not have control of the environment for can be used for cheating. In particular, you gave him the test to take home, and he passed it. In between when he left your office and when he came back, you had zero control of anything he did. Saying "he may have Googled the answers" is really the least of your problems; he could have whole-hog just given the test to his buddy and had his buddy write all the answers for him for all you know.

Here's a few options for what you can do if you are interested in this candidate:

1. Have him do a proper programming test, in a real programming environment. You set up the laptop, you set up the environment, you set up the IDE, he does the code. He gets full access to autocomplete if he wants it, a compiler if he needs it, etc. Then you can see if he can actually code (rather than write on paper). In real life, everyone uses efficiencies that modern IDEs provide, so there's no reason to not do so in an interview.

2. Invite the candidate back, and stress to them that they can use pseudocode; if they write "List.append" instead of "List.add", or even "add X to list", that's fine. Again, in reality in a job situation, they will have access to things to do this for them. Perhaps the candidate is hung up on "did I get the right syntax?" and can't give you what you actually want. If you say syntax doesn't matter, then you might get what you want.

3. Ask probing questions. "Write FizzBuzz" doesn't actually tell you much; it just tells you if he has memorized the solution to FizzBuzz. "Explain to me how you would write a program to do X" tells you a lot more. Specifically, it tells you if this person has the mental aptitude to design a solution to a problem that they may or may not have seen before. If they can't even tell you, in plain words (and "tell" you is an operative word, this should be a verbal exercise, to avoid issues such as dysgraphia or dyslexia), what the first step to solve the problem is, that's the red flag you're looking for to send them home.

Answer 10

He could have DYSGRAPHIA like I do. Or autism, or anything else that might cause trouble in writing things down.

Aside from that, unless you intend to have people programming a three ring binder, ditch the paper tests. Grab a spare laptop and have the applicant program/test on that.

Additionally, we are not in the days where you had to pour over books and manuals and memorize everything. Programming languages have become far more complex, nuanced, and verbose than back in the days of COBOL.

Answer 11

It could be that the guy is very good with his tools, and very bad without them. Or it could be that the guy is simply a cheater, stealing on other's job, and just assembling whatever he can find without much understanding.

At the end, whatever the answer, all come down to your specific needs. do you need someone at ease with any kind of tools? Someone able to code an algorithm in COBOL even as if he never met the language, and hates procedural coding? Then however good this guy might be, he's not a good fit.

Do you need someone who is assembling code, because most of the hard work in your shop is done by APIs, and the hard part is to find the clever APIs and use them efficiently? Then this guy, how bad he might be, is actually a good fit.

Decide on your needs. Decide on why the paper test is relevant - or not. Decide on why the home test(with the possibility of asking for solutions on internet) is relevant - or not. There lies the answer. For many position, someone who asks the right questions on internet and is skilled enough to assemble the answers a meaningful ways is enough. For some others positions, the same guy is just a liability.

Rethink about what you really need, and make your decision accordingly.

Answer 12

He requested to take the questions back for answers, which I agreed. The next day he sent me the fully corrected solutions. He did the assignments on an IDE and managed to get all of the answers correct.

Two things probably happened:

1. He googled the answer. Probably took it right off stackoverflow. I forgot who wrote the article but I remember one person mentioned the more elaborate/clever their FizzBuzz is, the more likely you know they googled the answer rather than thought of it from the top of their head. The assumption is people who understand programming wouldn't need to look up something as simple as FizzBuzz and would come out with the simplest answer.
2. He thinks better on the IDE than he would with a pen and paper. He could also rely upon the IDE auto complete or lookup features like that in phpStorm or msdn integrated into Visual Studio.

I agree with Ertai87 answer in that the best way is to give them a real world test. Afterall, the purpose of the FizzBuzz test is to determine if at the very least they can program but that is a separate thing from getting actual work, going into code, and putting in the proper solution.

Answer 13

For me personally, I program rather well (I think); but I've never been good with terms or concepts or expalining things to people when asked questions about programming. It may have something to do with how I learned or how I think, I don't know. But it doesn't mean that I'm not a good programmer.

On the other hand, I have seen a number of programmers who can explain all the concepts, know all of the terms, and seem really legit until they get down to coding and it turns out they are pretty bad at it.

Which kind of programmer would you want, someone who can code good or somone who can talk good about coding?

Also, I can't remember sh!t wihout intellisense. Do people really think that programmers can remember every method of every type of every language they're supposed to work with?

Also, tests are stupid. FizzBuzz is stupid. Give me an assignment that I can do in an IDE, like how I would work if I were working... and it really doesn't matter how I do it if I get it back to you and it is what you want. I can use google, stack overflow, c# corner... what does that matter? A non-programmer can't program by looking at stack overflow. A non-programmer wouldn't even know the right questions to ask to get the answers they're looking for.

You really should just go with your gut, which I'm sure is telling you to hire him.

Answer 14

There is a possibility that he did cheat in University and his internship was him working for a family friend in a high position. I know of a friend who gets good internship, for however long she wants per week due to her mother's connections. I also know of someone who cheated their way into University, and then had one of the years at the university done by someone else.

I'd be very cautious here, provide another test with a time limit over email and let him know he will get this at a certain time, and has this amount of time to finish it. If she/he is really all that they could also do something much more difficult than entry level programming.

Answer 15

Give him some more tests, but use a whiteboard this time so it's not so different from an IDE. If he still can't do it, pass him up.

Paper tests just check for over-learning - being able to quickly recall facts that you've memorized so that they're usually in your working memory. If you're good at development there's no reason to know more than motifs because any additional knowledge that you need can just be looked up in Docs. You should spend your brain power learning new technologies, not syntax and basic stuff you just don't use. A lot of devs I talk to haven't touched stuff like sorting algorithms since college. Paper tests should stay in college where they belong.

Fizz buzz on a whiteboard is a lot easier to solve because you can just erase and make adjustments on the fly. You can't always do that with paper. Design specifications also change a lot. Perhaps you want to be able to see your dev do something else, or perhaps your dev just literally did the easiest thing they possible could have. On paper you'd have to make a judgement call about docking points. On a whiteboard you could just change requirements on them and see how they respond without really worrying about anything.

Answer 16

there is a chance he is a google-smart programmer :)

https://www.hanselman.com/blog/AmIReallyADeveloperOrJustAGoodGoogler.aspx