23

I have an online programming test for an entry level analytics consultant role. I have always referred online code or documentation during programming but in the test I am not allowed to do so. Now I don't know what's the point of testing memory if I can code with the help of google.

What's the point of have proctored programming tests if one can do well with the help of documentation or online code snippets? Is it fair to test like this? How can I convey that I can code well it's just I am not good at remembering all the stuff so that I don't have to look up online?

I could understand if it was a basic programming test but it's moderate to hard level programming test which in real life even experienced professional would take help of documentation or google.

Note the test will be on hackerrank platform and I am not sure if I logic or implementation will matter if I don't pass all the test cases.

Update:

Before the start of the test it has a check box with note: "I will not consult/copy code from any source including a website, book, or friend/colleague to complete these tests, though may reference language documentation or use an IDE that has code completion features."

How would I use IDE or documentation if I am not allowed to escape full screen? Should I explicitly ask what's allowed and what's not allowed?

  • 2
    Could you give more background on the test? I have doubts that the test is truly "moderate to hard level" for an interview with no google. – Josh Jan 7 at 16:16

11 Answers 11

53

Is it fair to test like this?

Is it sensible? I don't know.

Is it fair? Yes, it's fair. If the others are asked to do the same thing. It's fair.

After all, the point of a test is not necessarily to ace it. It's to compare your performance to the performance of others. In that sense, it's fair.

If someone used to code with google and someone can just remeber everything, such kind of tests are not fair. They test more ability to remeber many things rather than programming skills. – Andrei Suvorkov

That's not exactly true. They don't just test the memory of the candidate. They also test the ability of the candidate to ask questions from the interviewer.

Besides, programming skill is in part remembering many things. The better programmers, the ones that have a ton of recent experience, are going to remember more. It's just the way things are.

It's just like with chess. The better chess players will remember board positions much better than a complete beginner would.

The more you practice at something. The better you become at organizing and clustering that information into your memory. Also, the more you practice something, some of it may even make its way into your muscle memory.

In any case, I'd suggest that you practice using the spaced repetition technique. The less you have to rely on Google for every little syntax question (especially of basic syntax), the more efficient you will become as a programmer.

That's really where you should place your effort because you're really not going to change their minds. Or the other alternative is to interview with other companies that are less stringent in their technical interviews.

And the day of the test, the point is not to get flustered. Use made-up function names or pseudocode if you need to. Just tell the interviewer what you're doing. Don't be afraid to say you don't know something and to ask questions when you need to.

And if you need to practice interviewing in real-time, I'd suggest you try these: http://pramp.com/ and http://interviewing.io

  • 8
    If someone used to code with google and someone can just remeber everything, such kind of tests are not fair. They test more ability to remeber many things rather than programming skills. – Andrei Suvorkov Jan 7 at 12:03
  • 4
    @AndreiSuvorkov say you are supposed to calculate mean value of an array foo. They who 'just remember everything' must know almost whole MATLAB documentation to write mean(foo,'all'). They who just forgot that such command exist may write sum(foo(:))/numel(foo) or a function based on even more primitive functions. – Crowley Jan 7 at 14:38
  • 6
    I've had to go in and clean up far too much spaghetti code that originated from a google search. – Richard Says Reinstate Monica Jan 7 at 19:27
  • 16
    As someone who has used over 10 languages in the same day/week, I find it hard to remember how to use commonly used methods, like string replace, even though I have +7 years professional experience and +25 years total programming experience. These test are geared towards specialists in a few languages/frameworks/libraries, whereas I've used over 35 and can't always remember what language needs what params or format for everything. – computercarguy Jan 7 at 20:34
  • 2
    @MandisaW, as an "established dev", I have to look for a new job way more often than I like, and if I don't look for a job in a variety of languages, the pickings are slim. If instead of just looking at C# jobs, I include Java, PHP, JavaScript and even C/C++, I have a much better chance of getting an interview, let alone a job, even if my C/C++ is rusty. I can always pick it back up, but a closed book test isn't going to show my programming skills, and doing the test in PHP isn't likely to show what I can do in for Arduino programming in C++. – computercarguy Jan 7 at 22:58
83

To paraphrase Cracking the Coding Interview: Most companies are aware that their tests will result in some false negatives. Particularly at the bigger-name companies that get hundreds and hundreds of applicants-- they're fine with that. What they want to avoid more than anything is false positives. A false positive means that they're wasting their money on someone who isn't going to help, who can potentially introduce more bugs to the codebase than a qualified candidate, and who ultimately costs the company more than they bring to the table.

In short: You're absolutely right. This isn't a real-world scenario, and it's testing you beyond what should be expected, especially of a junior candidate. But that's not really their concern-- their concern is to make sure that they get the best candidate least likely to make any errors.

edit: Here's a proposed scenario. We have three companies with identical holdings and reputations in the market. Allare seen as very attractive job prospects.The exact same pool of 10,000 candidates applies to each company.

Of the 10,000 candidates, 100 of them are qualified and excellent candidates. 9,900 of them are not suitable for the position, and could potentially damage the company's infrastructure.

Company A decides that it is critical that they look at all of the candidates and pick out only the ones that they want. In round 1 of interviews, they use a basic FizzBuzz test, and they knock out 5,000 right away. In round two, they send out an interview question that's at about the level of difficulty that they expect for the job, and they narrow it down to around 300 candidates. They conduct in-person interviews for those last 300 candidates, and they narrow it down to the 100 qualified candidates. They compare those candidates to one another, and they select their favorite.

Company B decides that they really only care about two things-- hiring quickly and getting a qualified candidate. They send out an impersonal FizzBuzz complete-by-email test to complete over email, and they weed out the same 5000 people who didn't know what they were doing. The second round of interviews requires bitwise operators, bit shifting, and all manner of niche, interview-only questions that 99.999% of the people in this role would never have to deal with.This narrows their search down to 23 candidates. Company B is confident that any of the 23 candidates will do the job well. They can interview them individually to find a good personality fit, then move on to the actual work.

Company C just sends out a basic FizzBuzz quiz, and they have 5000 candidates that they decide are all good fits (many false positives). They select one at random, because time is money, darn it-- I'm sure they can all do the job! They realize after the full on-boarding process, getting benefits set up, etc. that their candidate just googled the answer to the FizzBuzz quiz and does not know anything about the relevant duties of the job.

Companies A and B can eventually arrive at the same final point of selecting a single qualified candidate-- but company B has done it in significantly less time, and at significantly less cost of manpower to search through candidates. Yes, they booted out a few people unfairly-- but especially for something low or mid-level, they don't need a brilliant developer. They just need a developer-- someone who can perform roughly as good as the next guy.

When the end of the year comes around and companies A, B, and C have to compare against one another, if all else is equal, Company B will have retained more money and had more time and energy to devote to projects than Company A. Company C, who selected a false positive, now has to go back to the hiring process, having lost both money AND time. It makes good business sense to weed candidates out with minimal concern for false negatives so long as you can still retain a reasonable number of true positives.

  • 24
    The issue of the false positives > plus 1. – Solar Mike Jan 7 at 12:18
  • 5
    yep, the risk associated with hiring someone you shouldn't is far greater than the risk associated with not hiring someone you should. – Richard Says Reinstate Monica Jan 7 at 19:30
  • But this test seems unrelated to whether the person will be a positive or negative, as it's not really related to day-to-day work. If positives and negatives fail this test at the same rate, it does not help in selection at all - you might as well flip a coin to see who gets to the second interview. I don't see why you'd want a test that all negatives fail and many positives fail, when you could design a test that all negatives fail and only some positives fail. Why is this test in particular any better for weeding out the negatives than a more practical exam? – Nuclear Wang Jan 7 at 21:21
  • 8
    @NuclearWang because this IS a test that all negatives fail and some positives fail. If you're willing to put forth the required effort to be prepared for such an absurd interview, then they're pretty sure you'll fit. Look at it this way. I have 10,000 applicants. Of those 10,000, 100 will do the job roughly equally well, and 9900 will fail it. If the 100 will do the job equally well, I don't care if my test fails 75 of them; I only need to find 1 of those 100, and this process saves time and $. – NegativeFriction Jan 7 at 21:31
  • 4
    +1: My brother-in-law works at Google, and he has said the same thing numerous times. The only priority is keeping bad people out, there will always be more good people applying for these kinds of positions, so rejecting a great candidate who had a bad day or a slightly different expectation doesn't hurt the company at all. – GreySage Jan 7 at 23:23
19

This test is OK. They do not want to test your ability to search in internet about the things. They want to test your way of thinking. Because programming is to understand the requirements and implement algorithms (IMHO).

Errors in the source can be corrected later but if you can't think like programmer you are (probably) not applicable for this position.

Even if remember some structures, codes this will not help you much if you can't understand the things.

And if all the candidates are tested under the same rules, yes, the test is fair. It's another story if this is good test (and good to whom).

  • 4
    They do not want to test your ability to search in internet about the things. - Well I would say this ability is also essential in our job. – Andrei Suvorkov Jan 7 at 12:07
  • 1
    @AndreiSuvorkov, yes, it is important. But not so much as thinking :) – Romeo Ninov Jan 7 at 12:07
  • 2
    @RomeoNinov your point is clear. But as I understood applicants should submit their code to the Hackerrank and there code will be verified by machine. It doesn't care how you are thinking, just if all tests were passed or not. Later probably human will look into the submition(successful), but who knows.. – Andrei Suvorkov Jan 7 at 12:10
  • 1
    @Stupid_Intern So how do you filter out someone googling the exact solutions to the test? Googling isn't bad; but you shouldn't be completely reliant on it. If you can't figure out a solution to a toy problem with a compiler at hand without Google, I probably wouldn't want to hire you. I have no way to get even a glimpse of your programming ability (if anything, it's showing you have issues). Libraries and frameworks are fine, but you should really also be able to use simpler constructs to solve the problem if necessary - if you don't remember the right class to use, just write your own. – Luaan Jan 8 at 9:02
  • 1
    @Luaan I think at the very least the terminal should be able to reference the manual or help for syntax or example of usage. – Stupid_Intern Jan 8 at 9:04
8

I have to stress that I'm not a fan of tests under these sorts of conditions, they aren't a particularly good test of a candidate's real ability since it's rather rare not to be able able to reference google or similar resources while carrying out real work.

That said it doesn't necessarily mean the test is unfair - if all candidates are subjected to the same test conditions then it's fair. If you want the job you need to take the test to the best if your ability, same as everyone else. If they want to hire someone with good recall for this sort of knowledge then that's their prerogative I guess. Just as you are free not to work somewhere if you don't like their hiring practices.

  • 6
    I have seen candidates crush the test, but come into the real world and not be able to code their way out of a wet paper bag.... – Mister Positive Jan 7 at 13:22
  • 1
    @MisterPositive That's more the fault of too much reliance on tests (and test-prep) rather than thorough interviewing. – MandisaW Jan 7 at 22:42
  • @MisterPositive How do such people explain how come they do so poorly "in the real world"? Perhaps the test wasn't testing the abilities it should've tested? – Rosie F Jan 8 at 8:31
  • @RosieF Seeing plenty of candidates google the exact question in the test in live interview over Skype, I'm not surprised. Tests usually have to do a quick estimate of how much real programming work the candidate has done. You can quickly tell if someone was working with e.g. Windows message loops or AJAX through one question. The same question can often easily be answered with a quick google search; but that makes it entirely useless for its intended purpose - you're not seeing whether the candidate can solve this particular problem - just how he would approach a similar problem. – Luaan Jan 8 at 9:05
5

Is it fair to test like this?

With any part of an interview; it's perfectly fair so long as you are not being singled out. Whatever a business thinks will give it the information needed to make an informed hiring decision is fair for them to ask of you.

Always worth remembering though; you are always free to decline any part of an interview - accepting the likely outcome that you will be dropped from the process at that point. Interviews are as much about you wanting to work for them, as them wanting to hire you.

What's the point of have proctored programming tests if one can do well with the help of documentation or online code snippets?

While speculating on some other company's motives is always a guessing game. Some likely reasons can be:

  • They think that it will give them an understanding of your core knowledge

  • They think that if you have worked in the language for a considerable amount of time, you will remember a considerable amount of how it works

  • They want to see how you work in an unfamiliar environment, or in a field where documentation doesn't exist (e.g. when working on the bleeding-edge)

  • This was the way they were interviewed, and so this is simply the way you will be interviewed

  • They are wary of hiring a candidate that passes the test by cobbling together existing results online, without having the requisite knowledge - whether this is actually a real problem or not.

How can I convey that I can code well it's just I am not good at remembering all the stuff so that I don't have to look up online?

Although this will depend on the employer, if they are willing to actually read your code (they should), rather than pushing it through an automated "what percentage did they pass" system - you are always able to use pseudocode for the areas you know how to solve but lack the syntax knowledge to complete (or where you need to make assumptions about how a certain API works).

For example, if you know you "there's a way to send a fake key-press" but don't know how to do it in the actual Win32 API without looking it up, you may write code along the lines of:

if(realLogic){
    SendWindowsKeyPress("ENTER_KEY"); // There is a way to do this in the Win32 API, but I do not have the reference to hand. Placeholder function signature used here.
}

Noting that comments are valuable, and explaining things you would need to visit a reference for - is perfectly valid.

Of course, if they aren't willing to read your code or have hard guidelines on "it must actually run or we won't hire you" - you are simply out of luck, and write it off as a company who you'd probably not want to work for in the first place.

  • 4
    This. I had an interview for a software engineer job and I have very limited programming knowledge. I wrote the code to my best ability (it definitely wouldn't run) they asked me to talk through what I'd done and explain why. Long story short, I got the job. They were looking for the logic and core understanding to be there – Bee Jan 7 at 13:13
3

I'd like to piggy-back off what @Romeo Ninov said in his post:

They want to test your way of thinking.

This is 100% correct. I recently had a tech interview via video conferencing where I was asked to solve two coding problems. I was allowed to use my language of preference.

I couldn't remember the syntax for doing a particular thing in this language and asked the interviewer if it would be okay for me to quickly Google it. They said I shouldn't worry about it and that's okay to just mock it out/use pseudocode in place of whatever syntax I couldn't recall. They also explicitly justified this by saying they were more interested in seeing my thought process, and not necessarily my ability to 1) solve the problems correctly, and 2) recall language semantics.

But that's obviously a little different than using Hackerrank. For that, you simply need to practice until those types of problems become second-nature.

(Or until they become obsolete as a weeding mechanism—we can dream, can't we?)

3

Offline references and manuals

Depending on what it is that you actually need, are you allowed (and able!) to use offline reference manuals for API functions, parameter names etc? In many cases those would be even integrated in IDE platforms.

That should be reasonable, and that should be enough.

  • yup. Though I've had one such programming test where all that was available was VI, not even a compiler to try to compile the code. Of course they didn't care whether it actually compiles, just if the code structure and algorithm chosen make sense and if it was cleanly laid out with halfway decent comments. – jwenting Jan 8 at 4:48
2

This test is not designed to assess your ability to google for solutions or ask questions on stack Overflow. It is designed to analyse how you personally address a problem without aid. You are expected to make errors. You are expected to fail.

What they actually want to see is when and how you fail. What type of errors you make. This way they are about to compare all the candidates and choose the one that fails with the least collateral damage.

Think of it like testing materials for constructions - like bridges, powerplants etc. Those materials are subjected to conditions that are far beyond the conditions of standard operations. You are forced to work under hasher conditions than usual so your strengths and weaknesses can be seen easier.

  • 4
    You are expected to make errors. You are expected to fail. - Not when you have to submit your solution on Hackerrank with automatic tests verification. – Andrei Suvorkov Jan 7 at 13:27
2

A coder who understands what he is doing might be slower without a web search, but one who can’t complete the job without cut-n-paste will before long paste the wrong thing and not understand it. I think this company may be onto something.

  • P.S. I intentionally didn’t use the name of NSA’s biggest competitor—the one that tells you what they think you want to see instead of what you asked for. – WGroleau Jan 8 at 4:29
  • you mean amazon? :) – jwenting Jan 8 at 4:49
1

The employer knows the job and what skills you need to do the job, so I think it's perfectly reasonable.

I understand what you are saying but rather it's part of a broken hiring process rather than an unfair one. Many recruiters I have spoken to agree.

The employer needs to know what you will do if someone else hasn't already solved the problem. For this, online technical tests don't work as they only give a pass/fail and don't allow for outside the box answers. They need to give you a real world problem and look at your approach.

An online technical test just checks your memory of syntax and arbitrary programming questions that are purposely obfuscated in a way no one would ever write code. - I would like to know when this replaced having a portfolio and good references.

So I understand and partly agree, but also see it from the employers point of view. Maybe suggest that instead of taking the online test, you suggest meeting face to face and answer any technical questions they have. - This is what I do. I refuse online tests and i'm happy to walk away from any job that isn't flexible enough to accommodate, but I know not everyone has that luxury.

1

You didn't mention if they require code that would actually run, or one that would not necessarily run as is but would just show them how would you solve the task.

I guess it's the second option. The point is to show your software designing abilities and it doesn't depend if you spell a library function correctly.

They might be also afraid of you using Google in an unfair way (cheating, copying whole solutions/getting help from people).

PS: They probably also don't insist on memorizing library functions (that wouldn't make sense): if you don't remember library function, just call a hypotetical one.

Not the answer you're looking for? Browse other questions tagged or ask your own question.