27

tl;dr: How to deal with programmers who are highly recommended, but cannot handle paper test programming question

I interviewed a candidate for a programming position. He is a recent graduate from a college in computer science degree.

He has an excellent recommendation from his supervisor while he was interning for a programming job, and he had a string of programming side projects under his belt during his university time. Not very difficult kind of projects, but definitely self-contained and usable websites/mobile apps. Based on the resume he looks like a stellar fit.

The only problem is that when I gave him a paper test-- basically he has to use pen and paper to answer some programming questions, he struggled and couldn't get a single one correct. My questions all are very basic programming questions-- somewhere along the level of fizzbuzz, matching an element in the array-- the kind of questions that can be easily answered by those who take one semester of entry level programming course.

And I already mentioned that I don't care at all about the syntax/language. Pseudo code is good enough for me

He requested to take the questions back for answers, which I agreed. The next day he sent me the fully corrected solutions. He did the assignments on an IDE and managed to get all of the answers correct.

This puzzles me-- if he is as good as his referral recommendation implies, why he can't handle the paper test? Is it possible that there are some people who can code well in front of a computer screen, but when comes to actually writing programming solutions down on a paper, they struggle?

Edit:

  1. I don't care about the syntax, this was communicated clearly to him
  2. The questions level only make use of the most elementary constructs like for, if... nothing at all about esoteric library calls. Anything that requires working knowledge in a certain framework is considered not-basic. So I am not asking the test taker to memorize any calls to http method.
  • 1
    Comments are not for extended discussion; this conversation has been moved to chat. – Jane S Jan 12 at 22:20
  • 12
    Why you would still be using a paper test is beyond me. When does anyone code on paper? You should have let him do the test in an IDE on a system with a mirrored-display, and watched his process. That will tell you MUCH more about who you're working with. Your test is equivalent to putting someone in front of the paint mixer at Home Depot and using that result to decide if the person is a good portrait painter. – Wesley Long Jan 14 at 16:47
  • At a minimum you'd think you could give the candidate a text editor and a computer, not just pencil and paper. Editing on pencil and paper is a pain. We programmers are used to being able to insert things. – user1602 Jan 14 at 16:56
  • 5
    As a programmer, I never believed in those tests. Too often they contain questions about stuff you never ever need to use and expect you to know for the sake of knowing things. – Danny Coulombe Jan 16 at 16:26
  • 2
    I have trouble determining what you are seeking from this question - which is why it isn't getting the attention required. Can you restate at the top in clear language what you are seeking from the internet please? – Mirv - Matt Jan 22 at 13:26

16 Answers 16

65

Is it possible? Sure. People can get so accustomed to an IDE with syntax autocomplete and syntax highlighting that they hit a mental block if they try to write code on paper or on a whiteboard.

Is it possible that he had someone else do the work? Sure.

Why not ask him to come back and do a similar set of problems with a computer with whatever IDE you want set up. If he's able to produce quality code in an environment that is much closer to the actual workplace, great. In the future, you might offer candidates a choice between using an IDE or doing the questions on paper up front if people have a mental block with paper tests.

  • 41
    +1 or people can have disabilities like me. I have dysgraphia, so I actually can't write much at all. (It gets lost between my brain and hand) but I can type very well. The further removed a test is from the job, the less valuable the test. – Richard Says Reinstate Monica Jan 11 at 14:23
  • An onsite coding session or code test where they get a computer and have to code something on the fly would be good, then have them explain what they did. This seems a better reflection on how people work as well. If you use pairing in your work you can have them pair with an employee as well. – Bill Leeper Jan 14 at 23:58
  • @RichardU if you have dysgraphia, then how did you pass school exams? – Graviton Jan 15 at 7:16
  • 1
    @RichardU While I partly agree that it's good to take tests close to real requirements, I'd consider discussion of pseudo code an essential part of most team based programming jobs. Now, in being asked to write code on a paper, the interviewee might indeed be in the mindset of "writing a program" instead of discussing the general algorithmic solution to the problem. In line with the answer I'd at least give him other options to explain how he would solve it: white board, me writing him explaining, talking etc. but the general ability to abstract away from code I'd consider an actual asset. – Frank Hopkins Jan 22 at 10:21
  • 1
    @RichardU I saw your original comment. You'd probably do fine with explaining an algorithm verbally then and if you feel like you can disclose that at the interview I'd certainly either do any writing on a white board for you if needed or let you use a computer/tablet to take notes. My only point was that coding alone isn't the only skill required from a software developer in many jobs, in particular being able to explain the how on an abstract level is an often valuable skill as well and I find it fair to test for that (but with open tool set). – Frank Hopkins Jan 22 at 12:35
40

This link from the comments https://blog.codinghorror.com/why-cant-programmers-program/ posts an interesting problem, one that is frankly frustrating to write on paper.

Why?

Here is the requirements in text:

Write a program that prints the numbers from 1 to 100. But for multiples of three print "Fizz" instead of the number and for the multiples of five print "Buzz". For numbers which are multiples of both three and five print "FizzBuzz".

Here are the requirements as steps

  1. Loop from 1 to 100 and print the number
  2. Check for multiples of 3 and print "Fizz" instead of the number
  3. Check for multiples of 5 and print "Buzz" instead of the number
  4. check for multiples of 3 and also multiples of 5, print "FizzBuzz"

This is where it falls apart. Programmers don't think how to do all 4 steps at once, they do one step at a time. On a computer this is easy, write the first step, then edit in the second, then edit in the third, check if the 4th worked.

But on paper, you would have to write it 3+ times, either on paper or in your head. This takes a huge amount of time because this isn't how programmers have been trained to think. And any small mistake is another rewrite.

Programming is an art of making 100's of small mistakes quickly and fixing them. Duplicating this in writing takes a ridiculously long time. A senior developer taking 15 minutes is completely reasonable.

This is also lot of stress to pile on during regular interview stress, an activity that is typically already difficult for programmers. So they freeze up.

  • 6
    This is a very good point as breaking down the requirements into steps is a good idea. In fact someone might write code and a test for each requirement which would make it even more challenging to do on paper. – Joe W Jan 12 at 0:15
  • But this is where allowing pseudocode takes away the need for rewrites. You could write on paper: LOOP 100 TIMES then indented: CHECK MODULO 3 and indented IF TRUE PRINT "Fizz" etc. The syntax in Python for example allows this without rewrites even on paper :) – Juha Untinen Jan 12 at 11:57
  • 3
    You didn't write down the requirements correctly ;) – Chan-Ho Suh Jan 12 at 23:51
  • 1
    @Juha Untinen does pseudo code prove anything useful though? Does the question prove anything beyond that the person being interviewed can't switch tracks and see the whole problem in their mind? Because that's a very specific requirement, usually for leads that skill is helpful. Now if i was hiring a front end Web developer, i might ask them to write a basic webpage with a button, a title, and a bold label just to see if they know anything at all. But that doesn't require any logic. – Trevor Jan 12 at 23:58
  • 1
    I somewhat agree with the principle described here (pen and paper is nothing like an IDE and nothing like the job being tested for) and would agree with it for a problem of any significant complexity. But the example of FizzBuzz is really, really, really easy and the assertion that "a senior developer taking 15 minutes is completely reasonable" is insane. – BittermanAndy Jan 14 at 23:04
12

I wrote my first programs in 1967, and got a programming job in 1970. For the first few years of my career, I wrote my programs out on coding sheets.

There are three significant differences between then and now, any of which might be a problem for a programmer with only IDE experience:

  1. Remembering outer structures. In an IDE, you can set up e.g. an if-then-else before filling in the content. On paper, you need to remember where you were with each incomplete structure.
  2. Looking things up: IDEs offer choices and word completion that are not available on paper. When programming on paper I needed reference manuals within reach for anything I did not have completely memorized. If you must do paper programming tests, provide the appropriate references either on a computer or on dead trees.
  3. Handwriting. I could produce legible block capitals very fast when I was doing that all day every day. When faced relatively recently with some academic exams that had to be done paper and pencil, I prepared by practicing writing, including writing mathematical notation. It was many years since I had done much handwriting.

Unless you really need the ability to program without an IDE, you should set up a programming environment for your tests.

8

if he is as good as his referral recommendation implies, why he can't handle the paper test?

Because there is no need to remeber everything you can google in no time. I find questions about algorithm complexity, finding array elements or sorting etc. intimidating and unprofessional. These are things you can read about very quickly if you need them.

Is it possible that there are some people who can code well in front of a computer screen, but when comes to actually writing programming solutions down on a paper, they struggle?

Yes, it is possible because it's much more important to have a general idea about what you are doing and being able to think abstract, take the long view or see the big-picture and be able to find the necessary information when needed than know by heart every algorithm. Pure encyclopedic knowledge is worthless when people don't know how to apply it properly.

Too many interviews focus on such things as whether you know how fast a certains sorting is (which you can find within few seconds) rather than to evaluate how people think.

  • 4
    So if asking you to work through a fundamental conceptual problem in computer science free of any particular language or framework isn't an opportunity to 'evaluate how you think.' what is? :) – Affe Jan 11 at 22:16
  • @Affe mhmm... I don't think we're on the same page yet because that's not quite what I'm writing about. I mean that how you think isn't being evaluated but your encyclopedic knowledge - which in most cases doesn't prove anything anyway. – red-shield Jan 12 at 9:53
  • I apologize that my trigger about ppl who answer every question with “I would just google that” came across as directed at you ;). My interns are great googlers, unless they want an intern salary give me something. I understand what youre saying about problem solving skills vs trivia. I don’t care if the answer is “right” or “wrong” for any one specific question, but interviewing has been getting frustrating in the past few years when I can’t find any topic people will actually have a conversation about that isn’t trivia about whatever framework they used most recently! – Affe Jan 13 at 18:00
  • There is article about Stack Overflow Driven Development - dzone.com/articles/… – Justas Jan 13 at 22:00
  • 1
    I have exactly ONE big O question in my usual interview question set, but it is followed up by a question about how would the candidate expect that algorithm to play on a real modern pc processor? What I am looking for is some awareness of branch prediction, the horrible cost of cache misses and similar things. Does the candidate understand that big O is not the only thing that matters. It is scary how many candidates either cannot answer the first bit or fall down badly on the difference between CS theory and real machines. – Dan Mills Jan 13 at 23:59
6

if he is as good as his referral recommendation implies

That's because your expectation is too high for both the candidate and the supervisor. The recommendation letter, if unbiased merely mean the candidate's level is satisfactory relative to the person who wrote the letter. That doesn't mean the candidate is skilful relative to your standards.

For example, an 14-year old kid who can compile a hello-world program may impress his grandmother, but not you.

You probably have better technical knowledge than the supervisor, or at least someone who has higher expectation in excellence.

5

Is it possible that there are some people who can code well in front of a computer screen, but when comes to actually writing programming solutions down on a paper, they struggle?

All sorts of things are possible from mental block to cheating. But none of them can be known for sure and none apply to what you need to do which is gauge competence and suitability. All else being equal this candidates CV should be at the bottom of the pile.

Reccomendations are just paper, better than nothing but not to be relied upon fully. I've seen glowing recommendations given to people that the recommendee barely knows or doesn't know at all.

4

Paper test are not bad and pretty much standard across industry in the area I come from. My suggestion would be to ask the candidate to write algorithms rather than actual code. I can buy that the candidate does not know the exact syntax. But they should be able to provide algorithms/ pseudo code.

There could be two potential issues

  1. Candidate may actually be cheating.

  2. Candidate will take more time to complete the assignment that usual , which means he spends his time in reading post the problem is given.

It comes down to the way you have been groomed. In our days, lot of focus was on algorithms and we were taught to write algorithms on paper. Before IDE's came into picture, we had to write code in notepad which made our hold on syntax much stronger than someone who has learnt on IDE. Getting syntax right is not very important, but if the candidate is struggling to a point that he/she cannot answer a single question, in my opinion, then the candidate needs to work on basics of programming. Chances are that they will struggle and take a lot more time to complete their assignment that what would normally be expected.

  • 2
    By the time there was notepad, there were IDEs. Unless you're talking about the VI editor in UNIX. Also, some of us predate GUIs, but a keyboard is still easier to put things down on than paper. Paper tests are obsolete – Richard Says Reinstate Monica Jan 11 at 14:20
  • 2
    Turbo Pascal was released in 1983, and was an IDE. Without changing programs, you could write a program, edit it, compile it, and run it. It was far easier to use than other development systems for home computers. Sometime around then, there was a system that would not let you write a syntactically incorrect Pascal program, but that made it a real pain to use. – David Thornley Jan 11 at 16:12
  • In my two degrees (Associate, and Bachelor) for software development, we didn't have a single course in algorithms. A lot of it was "embedded" into the subject, but never explicitly told that we are now studying algorithms. We never had anything like studying quicksort, doing btrees or O-notation and analysis. It was all about developing solutions to various problems using whatever method we deemed suitable. Mostly using C. – Juha Untinen Jan 12 at 11:48
3

"Is it possible that there are some people who can code well in front of a computer screen, but when comes to actually writing programming solutions down on a paper, they struggle?"

Absolutely possible. Different people work in different ways. Their brains work in different ways. And some people just have problems writing things down by hand.

I can type things without having to think about the typing process at all, it goes straight from my brain to the screen. Writing on paper is harder. (Speaking things is a lot harder, turning thoughts into speech takes a large percentage of your brain power).

So if you want to observe people, you observe them using the tools that they are used to, and there you see how well they can do. If someone is bad writing on paper and good using an IDE, hire them. The other way round, don't.

3

Anything the applicant could have done without you watching over him is of more or less zero relevance when it comes to gauging his performance. Anything you do not have control of the environment for can be used for cheating. In particular, you gave him the test to take home, and he passed it. In between when he left your office and when he came back, you had zero control of anything he did. Saying "he may have Googled the answers" is really the least of your problems; he could have whole-hog just given the test to his buddy and had his buddy write all the answers for him for all you know.

Here's a few options for what you can do if you are interested in this candidate:

  1. Have him do a proper programming test, in a real programming environment. You set up the laptop, you set up the environment, you set up the IDE, he does the code. He gets full access to autocomplete if he wants it, a compiler if he needs it, etc. Then you can see if he can actually code (rather than write on paper). In real life, everyone uses efficiencies that modern IDEs provide, so there's no reason to not do so in an interview.

  2. Invite the candidate back, and stress to them that they can use pseudocode; if they write "List.append" instead of "List.add", or even "add X to list", that's fine. Again, in reality in a job situation, they will have access to things to do this for them. Perhaps the candidate is hung up on "did I get the right syntax?" and can't give you what you actually want. If you say syntax doesn't matter, then you might get what you want.

  3. Ask probing questions. "Write FizzBuzz" doesn't actually tell you much; it just tells you if he has memorized the solution to FizzBuzz. "Explain to me how you would write a program to do X" tells you a lot more. Specifically, it tells you if this person has the mental aptitude to design a solution to a problem that they may or may not have seen before. If they can't even tell you, in plain words (and "tell" you is an operative word, this should be a verbal exercise, to avoid issues such as dysgraphia or dyslexia), what the first step to solve the problem is, that's the red flag you're looking for to send them home.

2

He could have DYSGRAPHIA like I do. Or autism, or anything else that might cause trouble in writing things down.

Aside from that, unless you intend to have people programming a three ring binder, ditch the paper tests. Grab a spare laptop and have the applicant program/test on that.

Additionally, we are not in the days where you had to pour over books and manuals and memorize everything. Programming languages have become far more complex, nuanced, and verbose than back in the days of COBOL.

  • 1
    My feeling is that things have got simpler, because what COBOL did with syntax is handled by library calls in modern languages. – Patricia Shanahan Jan 11 at 23:44
1

It could be that the guy is very good with his tools, and very bad without them. Or it could be that the guy is simply a cheater, stealing on other's job, and just assembling whatever he can find without much understanding.

At the end, whatever the answer, all come down to your specific needs. do you need someone at ease with any kind of tools? Someone able to code an algorithm in COBOL even as if he never met the language, and hates procedural coding? Then however good this guy might be, he's not a good fit.

Do you need someone who is assembling code, because most of the hard work in your shop is done by APIs, and the hard part is to find the clever APIs and use them efficiently? Then this guy, how bad he might be, is actually a good fit.

Decide on your needs. Decide on why the paper test is relevant - or not. Decide on why the home test(with the possibility of asking for solutions on internet) is relevant - or not. There lies the answer. For many position, someone who asks the right questions on internet and is skilled enough to assemble the answers a meaningful ways is enough. For some others positions, the same guy is just a liability.

Rethink about what you really need, and make your decision accordingly.

1

He requested to take the questions back for answers, which I agreed. The next day he sent me the fully corrected solutions. He did the assignments on an IDE and managed to get all of the answers correct.

Two things probably happened:

  1. He googled the answer. Probably took it right off stackoverflow. I forgot who wrote the article but I remember one person mentioned the more elaborate/clever their FizzBuzz is, the more likely you know they googled the answer rather than thought of it from the top of their head. The assumption is people who understand programming wouldn't need to look up something as simple as FizzBuzz and would come out with the simplest answer.
  2. He thinks better on the IDE than he would with a pen and paper. He could also rely upon the IDE auto complete or lookup features like that in phpStorm or msdn integrated into Visual Studio.

I agree with Ertai87 answer in that the best way is to give them a real world test. Afterall, the purpose of the FizzBuzz test is to determine if at the very least they can program but that is a separate thing from getting actual work, going into code, and putting in the proper solution.

1

For me personally, I program rather well (I think); but I've never been good with terms or concepts or expalining things to people when asked questions about programming. It may have something to do with how I learned or how I think, I don't know. But it doesn't mean that I'm not a good programmer.

On the other hand, I have seen a number of programmers who can explain all the concepts, know all of the terms, and seem really legit until they get down to coding and it turns out they are pretty bad at it.

Which kind of programmer would you want, someone who can code good or somone who can talk good about coding?

Also, I can't remember sh!t wihout intellisense. Do people really think that programmers can remember every method of every type of every language they're supposed to work with?

Also, tests are stupid. FizzBuzz is stupid. Give me an assignment that I can do in an IDE, like how I would work if I were working... and it really doesn't matter how I do it if I get it back to you and it is what you want. I can use google, stack overflow, c# corner... what does that matter? A non-programmer can't program by looking at stack overflow. A non-programmer wouldn't even know the right questions to ask to get the answers they're looking for.

You really should just go with your gut, which I'm sure is telling you to hire him.

  • 1
    May I ask what kind of programming jobs you've held? My experience is that if you are working on a team on a complex project you need to both be able to code and to explain what your code is doing to your team mates. – Charles E. Grant Jan 11 at 20:32
  • @CharlesE.Grant I didn't say that I couldn't explain what my code is doing. What I mean is that I can't explain the concept of OOP in any way that sounds like it's correct. Or what polymorphism is. And if you have to explain what your code is doing to someone else, you didn't code it right and/or the other programmers can't read. A good programmer should be able to look at good code and figure out what its doing. – Barry Franklin Jan 11 at 20:57
  • Although as a programmer I agree with your answer. The answer is very negative sounding like your venting your frustrations into the answer. – Trevor Jan 11 at 22:20
  • 1
    You need people who can code well and who can talk about coding, unless you're hiring one person and aren't going to hire another. Someone who can't express basic concepts is a lot less useful than someone who can. Also, tests like FizzBuzz are useful as a quick way of weeding out the completely unsuitable. – David Thornley Jan 11 at 22:36
  • 2
    I agree that programmers can't be expected to remember every method on every class in every library they use. But you don't need any libraries to write FizzBuzz. You need to write a for loop and use the mod operator. FizzBuzz may be stupid, but if you can't write a for loop off the top of your head, then you're not programmer. – Dan Jan 12 at 0:02
0

There is a possibility that he did cheat in University and his internship was him working for a family friend in a high position. I know of a friend who gets good internship, for however long she wants per week due to her mother's connections. I also know of someone who cheated their way into University, and then had one of the years at the university done by someone else.

I'd be very cautious here, provide another test with a time limit over email and let him know he will get this at a certain time, and has this amount of time to finish it. If she/he is really all that they could also do something much more difficult than entry level programming.

0

Give him some more tests, but use a whiteboard this time so it's not so different from an IDE. If he still can't do it, pass him up.

Paper tests just check for over-learning - being able to quickly recall facts that you've memorized so that they're usually in your working memory. If you're good at development there's no reason to know more than motifs because any additional knowledge that you need can just be looked up in Docs. You should spend your brain power learning new technologies, not syntax and basic stuff you just don't use. A lot of devs I talk to haven't touched stuff like sorting algorithms since college. Paper tests should stay in college where they belong.

Fizz buzz on a whiteboard is a lot easier to solve because you can just erase and make adjustments on the fly. You can't always do that with paper. Design specifications also change a lot. Perhaps you want to be able to see your dev do something else, or perhaps your dev just literally did the easiest thing they possible could have. On paper you'd have to make a judgement call about docking points. On a whiteboard you could just change requirements on them and see how they respond without really worrying about anything.

0

there is a chance he is a google-smart programmer :)

https://www.hanselman.com/blog/AmIReallyADeveloperOrJustAGoodGoogler.aspx

Not the answer you're looking for? Browse other questions tagged or ask your own question.