How to test a programmer's ability to handle a large code base?

Question

I would like to get advice regarding job interviews for hiring software developers.

A couple of years ago, our company hired a software developer. One of the shortlisted candidates was clearly above the others - or at least so we thought. This senior programmer was able to answer impeccably all the technical questions in the two interviews (which included writing short pieces of code on the spot), and his soft skills, too, were excellent (well-mannered, very good at communicating his thoughts, etc.). We hired him.

Unfortunately, he wasn't nearly as good as we had thought. He was indeed very good when given narrow, well-focused tasks, but was never able to grasp the bigger picture of our software. After nearly two years in our company, he decided to leave (and joined another company).

I wonder if anyone has faced the same problem. Can you detect the weakness of such a candidate? One problem with technical interviews is that they focus on smallish tasks, so you don't get an opportunity to see how a person can cope with a more realistic, bigger code base.

Answer 1

One way to design interviews to avoid selecting purely on short technical questions is to use more experience- or competency-based interview techniques. Here is a representative, but very general, link.

In software, a good way is to get the candidate to describe their current system. You can start small, and explore together, often in front of a whiteboard. Preferably the candidate would do the sketching. Feel free to add the reassuring caveat that they shouldn't share proprietary details. With better candidates, you will be able to get to a fair sketch of one of their recent systems within twenty minutes or so. A good technical candidate will often enjoy the process. You can also then stress test the design with hypotheticals, eg: what if we needed to break this assumption, what if we had ten times this type of message.

A similar, and complementary technique, is to give a design problem, like designing Facebook, or a stock exchange matching engine. This will be too large scope to solve everything within an interview, but a good interviewer can then cut the scope to focus on interesting parts of the problem.

In previous roles I worked in, we had a multi-stage interview process. The initial interview tested for competency at the short-solution, line of code, level. The second and third interviews were a chance to understand the candidate's thinking process when zooming out and choosing between design tradeoffs. Team compatibility and checking whether senior people were comfortable could also be explored in these later stages. Without the short coding problem screening, these more talky interviews can be a little more vulnerable to manipulation by good blaggers, and verbose timewasters. Checking they can really write code first avoids this.

Other experience-based questions for programmers include:

• What's the most difficult bug you fixed in the last two years?
• How would you respond when your most important customer calls you to say the system is broken? What are your short- and long-term actions?

Executing on an interview project is another way, but asks for a fairly big time commitment from the candidate, and is vulnerable to forms of plagiarism.

Answer 2

I think you are looking at it the wrong way. It is very unlikely that a person is good at everything except working with a large codebase. Instead of testing specifically for that, you need to change your approach to interviewing senior engineers entirely. As you correctly state, interviews that just contain small test-like tasks are not very good for that. Try asking more "why" questions instead of, or in addition to, writing the code. Try asking the person to review some code. Try a design discussion instead of LeetCode. And, of course, the interviewer should have the necessary experience.

Answer 3

ability to handle a large code base

and

impeccably [...] writing short pieces of code

See? That is your first hint. You had one reality, and you tested the candidate for something else.

---

Unfortunately, he wasn't nearly as good as we had thought.

Unfortunately for you, you are totally right. He was exactly as good as you TESTED him. But you THOUGHT wrong in the first place about the entire situation, and that is why you tested him for the wrong job.

Practically, you did what other managers did with me several years ago. They interviewed me for a software project management job, and the content of the tests was literally writing or correcting short sequences of code.

And the hings went downhill afterwards, because their bad management "style" did not stop at interviews. I was not allowed to produce any course correction (because that would break the "team"), and additionally the same managers were constantly taking the team off-course with all kinds of badly-understood "team building incentives".

And it seems that something similar happened in your case too. You noticed that you made a bad hire, but instead of looking at you for expecting the wrong thing, you blamed and (probably) harassed him instead. Probably something like "We had great expectations from you because you are senior, and you disappoint us". I've been there too.

---

Bottom line: the only real seniors in your code base are the people who wrote that code base. Anyone else is going to have problems. The internet is full with questions about "spaghetti code" and "unmaintainable code" (including the SE sites). So you understood your problem wrong, you attempted to solve the problem in the wrong way, and you expected miracles from a poor fellow.

Was he indeed suitable or not unsuitable? We do not know. Did you handle most of the details badly? Quite yes.

---

One problem with technical interviews is that they focus on smallish tasks, so you don't get an opportunity to see how a person can cope with a more realistic, bigger code base.

That is only if the technical interviews only focus on smallish tasks. Why did you not give him a more complicated task, relevant to the job?

Additionally: why was there no probation period, to understand if he can actually grasp the project? Again, the weak link was you, not the candidate.

At my current job, the interview had exactly zero questions and zero tests. I was hired actually based on the content of the CV. Later I asked the manager why was that, and the answer was simple: "That is why you are in a probation period. If I am not satisfied, you go." (the words were different, but the message was the same)

Answer 4

Just give them a test task to accomplish in a big codebase.

You don't need to show them your proprietary in-house code if that's what you're concerned about. Just pick a relevant open source project and give the candidate a task that is relatively simple to implement but requires orienting yourself in the codebase first.

For example, if you want to hire C programmers, point them at the Linux kernel and ask them to modify it so that all process IDs are always multiples of 13.

Of course they don't need to actually build and test it, but they should be able to show you the place to modify and write a five-line tentative patch. Then just have a short chat about their solution.

Answer 5

How to test a programmer's ability to handle a large code base?

You can't test their ability to handle a large code base.

You tested their ability to write small programs or parts of programs when given a prompt. Unfortunately that wasn't what you were looking for.

You need to ask them questions and start discussions about their previous positions. You need to understand if that history and experience shows an ability to handle a large code base as an architect or engineer.

You also didn't look internally for the best person on the current team or the best person in your company to take on that role.

This senior programmer was able to answer impeccably all the technical questions in the two interviews (which included writing short pieces of code on the spot)

A senior programmer is just a programmer with X years of experience. That doesn't mean they were ever asked to do anything more than write parts of programs based on prompts.

I have seen this mistake made many times. In once case the company wanted to bring on a person who had 10 years experience operating satellites for NASA. Unfortunately they wanted them to write the operations manual. That 10 years of experience was rooted in following the procedures that other people wrote, and never wondering why things were done that way. It didn't work out.

Answer 6

It’s hard to test in an interview. In the cases here it took a month to find out.

You can only ask the candidate: Have you done this before? Are you willing and do you feel capable of doing this? Because that will be your job. No point taking the job if you can’t handle this. Most people won’t be so desperate to take a job they cannot handle.

And then working with a massive code base is hard. It takes time. With a steep learning curve. Maybe you expected too much after a month. Maybe you should offer the person to be hired as a contractor for a month, and no hard feelings if it doesn’t work out.

Answer 7

Learn From Others

Obviously, every large tech company must hire engineers that know how to work in a large legacy codebase. So they face this problem every day, for hundreds to thousands of candidates. You should ask yourself how they solve the problem. The answer is pretty obvious and simple: they interview candidates at multiple levels of detail. Virtually every tech interview loop will include coding interviews. But for anything above an intern/college grad, there will also be design interviews where the candidate "codes at the architecture level", if you will.

Understanding legacy codebases first requires you to have an ability to see the system at a level of abstraction where entire applications are just "subroutines". The "variables" are databases and the "functions" are microservices. Ask the candidate to design a small system at this level, and see how well they reason through the important elements and features. If they can't do that, then they will never be able to grok your codebase.

A popular such question is: "Design Twitter". Where and how do you store the data? Do they understand distributed computing issues like the CAP theorem? Can they identify the essential functionality of the service while ignoring irrelevant details? Obviously, a candidate that spends a lot of time explaining how they would display animated emojis is probably going to be a hard pass. The nice thing about such questions is that they don't have a right answer, and so it doesn't matter if everyone knows you ask this question. Even if they rehearse an answer, it doesn't mean their prepared answer touches on all the issues that you think are important. The actual details of what they say aren't nearly as important as their reasoning which led them to a particular design decision, and that is what you should be probing with lots of follow-up questions.

Do Your Homework

Look at their resume. What is the largest project they worked on? Do they have any experience which implies they have even been exposed to a large legacy codebase? If they are new in their career, you wouldn't expect this. But if they have 10+ years of experience, not having any exposure to a large codebase is definitely going to put them behind the pack. It suggests they mostly worked for smaller shops and never had to solve problems with scale that they will face in your organization. It means if you take them on, there will be extra ramp-up that both of you must acknowledge and accept.

Push Them In...

...the deep end of the pool. Tell them you want to add a certain new feature to your system. Ask them to draw an architecture diagram of what they think your current system looks like. When you see major missing pieces, tell them more about how your system works and see if they can infer what needs to be added to make the diagram more accurate. When they are close, just fix their diagram in a generic way (don't use actual service names, just make something that gives the idea). Obviously, you will need to guide them at each step of the way, giving more information to help them reach the goal (you should always help the candidate reach the solution, for every problem, IMO).

You will look for two major indicators: first, you want to see how much they can piece together from just their knowledge of the public info about your company/product, as well as the hints you provide. The more they infer on their own, the better. The more hand-holding they require, the more hand-holding you will need to provide once they are on your team. Second, you want to see how they respond to the process emotionally. Are they excited to tackle a somewhat vague challenge that will require a lot of investigation and interaction to solve? Or do they get frustrated by the ambiguity and the high-level nature of the task? Their attitude while tackling this problem will likely tell you as much about their aptitude and suitability as their actual skill in solving it.

Answer 8

TL;DR - You can probably test ability/desire to dig through "muck" as part of screening or take home questions.

I am just not sure if people that you may want hire, will eliminate themselves because your company just isn't that good / desirable to work for....

---

I worked for a company that gave out a screening question where the actual coding problem was fairly trivial, but there was a lot of googling to find out how to use the libraries/frameworks that the question mentioned/required.

It wasn't my question, but part of my process is to personally solve all the questions given to candidates - so I can evaluate what potential challenges the candidate might face / get a feel for the complexity and any ambiguity in the questions. Doing so I found the question both irritating and interesting, interesting because I learn't a new (to me) library, irritating because what seemed like a trivial problem, forced me down a set of rabbit holes to work with the libraries. It took me about 45 minutes to come up with my answer, but it was a Java question and we were a Java shop, so I was already familiar with most of the stack.

The answers we got from candidates varied, some candidates totally ignored the library specifications and just solved the trivial problem, some tried to use the libraries, but didn't do a very good job and some fully embraced the libraries and used them correctly.

The on-sites that I took, generally matched the screening question - people tended to do well in both or not well in both.

However per my TL;DR - I don't know how many candidates aborted the process because they didn't feel the time investment was worth it.

---

PS I started this answer saying "muck", because if you have a large but pristine code base with amazing documentation, 100% code coverage with full CI/CD, well thought out requirements and clear stories it's far less likely developers will struggle.

Answer 9

This senior programmer was able to answer impeccably all the technical questions in the two interviews

First thing to consider: is your measure of a senior developer solely their ability to answer technical questions? In other words, what is it that, to you, makes a senior profile a senior profile?

Because when I'm hiring senior profiles, I barely ask technical trivia questions unless I particularly feel like they're oddly lacking in that experience. I'm much more focused on how they balance certain natural tensions, e.g. maintaining high code quality standard in the face of a product manager who wants to meet a deadline that's too tight; and I asked about their process of ensuring high quality and avoiding technical debt.

More importantly, I'm looking for a subconscious understanding that these considerations require nuance and balance, and not dogmatic adherence to "the right way of doing things".

(which included writing short pieces of code on the spot)

Building something from scratch is an entirely different skillset than inheriting an existing thing and keeping it both operational and further extending it without causing significant delivery blockers as part of the onboarding process onto the inherited system.

A much more valuable exercise would be to give them an existing (demo) codebase and ask them to fix a few issues, while at the same time giving them the open-ended challenge of identifying technical debt and drafting a proposal as to what they'd change moving forward.

Unfortunately, he wasn't nearly as good as we had thought. He was indeed very good when given narrow, well-focused tasks, but was never able to grasp the bigger picture of our software.

The straightforward truth here is that I have no basis to agree that the developer was unreasonably unable to handle your reasonable code base or development process. It's just as possible that the developer was reasonably unable to deal with an unreasonable code base or development process.

One example of this is that your question focuses on a developer's ability to "handle a large codebase", which subtly excludes the idea that things are easier to maintain when they are decomposed into smaller, individually more digestible components.
You say that the developer couldn't handle a large codebase, but how sure are you that if I ask that developer their opinion, that it would be that the company keeps working with an uncomfortably large codebase?

After nearly two years in our company, he decided to leave (and joined another company).

No mention of any formal processes for them not meeting their requirements + them leaving the company, suggests a possibility that they didn't fail you, you failed them.
They're the one who left you, but you're describing them as if they were the source of the problem that caused them to leave. That math doesn't add up.

Can you detect the weakness of such a candidate?

This phrasing, i.e. an implicit assumption that it was the developer who was defective and you who were in the right, very much aligns with what I have historically seen in companies that struggle to maintain technical excellence (for lack of a better term) in their codebase. I don't see anything in your question that counters this notion.

---

Overall, my educated read on the topics raised by your question is this:

• Your interview process is faulty as it does not accurately test for the qualities in an application that you will then judge an employee by.
• I cannot definitively judge who is right here, but I do sense a distinct lack of self-reflection as to if the company contributed to the friction between them and the developer. I would recommend a self-reflective moment to align if the expected work (i.e. the large codebase) reasonably matches contemporary coding standards these days - because I get a feeling they might not.
• If an employee does not meet your expectations to the degree that this question implies, then I'm wondering where the action you took is in the two years that they were employed and in your words never displayed an ability you consider to be essential. Either the question is incomplete or it is unclear to me what you were doing to address the issue during their employment.

Answer 10

When you have an objective, you should define them.

What exactly does "handling large codebase" or "grasp the bigger picture of our software" even mean?

If you give actual concrete examples of failure, then we can dig into the specifics, but this is so vague as to be meaningless.

It may indicate that this is what your expectations are to deliverables when some of the unknowns are not appropriately assigned.

Answer 11

As a 30+ year software engie, I wonder if your environment (your large code base) is adequately described and documented in such a way that a new coder can make sense of it. I'm good, but I've been in a situation where it took upwards of 2 years to understand how everything fit together because I had to learn it from experience and piecing together a mental picture one meeting at a time. Those who came before understand already so why bother writing a 101 doc?

Bottom line, if you're going to have a complex software environment then you need to make a roadmap... for even the best of us. -> You Are Here <-

Answer 12

I have encountered this. I gave the shortlisted candidates a mini-project to do at home, then I had an interview during which I gave them two small bits of code to write and two bits of code to find bugs in. I even asked them about the choices they made on their mini project. This particular one stood above the rest.

Within a month it was obvious that he was not suited for the team. He needed everything specified and written down. So, I had started specifying everything that was required.

As an example, in this sub-task, we discussed everything in a meeting and I bulleted the items that were required. I didn't specify what index a a new table needed. I felt that I would have demeaned him by specifying such an obvious item. However, since I didn't specify it, the table had no index at all.

As another example, a small change that was required to one unit, may have affected other units. I gave him a list of 3 as an example of units that may be affected, with an instruction to check all dependent units. So, he only checked the 3 units.

From my point of view, it was a failure on his part to think about what he was doing. I did not need an architect. But I didn't need a robot either.

Being in NZ at that time, I followed the law and had three meetings (one a month) with him discussing the shortcomings and giving him an opportunity to improve. He didn't. So, I suggested that he was more suited to a bigger organisation, he agreed. I gave him a good reference and we parted company.

I am surprised that your company waited so long.

Answer 13

Ask them - "How comfortable and confident would you feel working on an old code base with messy code?"

Enter a discussion on this point and see if they confidently can talk about working with this type of assignment before.

Reference: Ive been asked this before in a technical interview. I really appreciated that the interviewers were honest about this, and they also told me they had trouble keeping people due to the state of the code base.

And yes, they offered me the job (but I didnt take it).

Answer 14

I have a keen interest in people who work on legacy Java projects and have read quite a bit about their work. In some of these projects, the nerds who actively maintain the codebase only do so on very specific parts of the codebase. It gets to the level that you might call it hyper-specialised. These projects have completely matured so it is in a maintenance phase and only sporadically have the need for active development.

So if the codebase is indeed massive why are you expecting some person to be able to maintain it in its entirety? Your project may do well if you can focus workers on a more specific part of the project and try to do the macro-management yourself.

You may find that the smaller the part of the project the individual has to deal with the higher the quality of the small piece may be. The coder never could be able to grasp the bigger picture, why would you expect him/her to do so? That is the job of the project managers and team-leads

Answer 15

If we ignore skill of all programmers involved and just categorize how difficult it is to write software, then roughly from easiest to hardest:

• Designing small programs from scratch.
• Debugging/maintaining small programs written solely/partially by yourself.
• Designing large programs from scratch.
• Debugging/maintaining large programs written solely/partially by yourself.
• Debugging/maintaining large programs written by others.

The last one is by far the hardest and usually requires a veteran as you can only learn this with experience. Or otherwise you end up with plenty of bugs at every patch. Which you might do anyway, in case the original code base is badly designed, poorly documented or just the victim of years of "code rot".

Also, maintaining code written by others is usually not very fun. It is difficult, it is not creative work, it is depressing if the state of the code is bad, which is common. Very few want to work with that 8 hours per day, so as a manager set your expectations accordingly: you need a senior dev and you need them to do boring work for you. Yikes... You'll likely need to both pay them well and otherwise motivate them to keep them around for the long term.

---

The way to test someone's experience at maintaining big code bases is not to have them write code - if you do that you merely test their ability to at all function as a programmer, by checking their ability to perform the easiest task in my list above.

But if you are specifically recruiting a senior dev who needs to maintain a large code base, then ask them about program design. Or show them some code and let them comment on the design. Or development life cycle stuff. Debugging, coding standards, version control etc. Meaning that you either need to be an experienced dev yourself or have one present at the interview for this part.

Those who know nothing about program design are probably not suitable for writing large code bases from scratch - they'll make a mess - and they are for sure not suitable for the even more difficult task of maintain large code bases written by others. People who sneer at design, coding standards and general discipline are usually not anywhere near as good as they think they are.

---

Similarly, a red herring is programmers who constantly talk about "refactoring" their own code, aka throw everything away and do it correctly instead. They are probably not veterans but intermediately skilled. All programmers go through this phase when they are reasonably experienced: they suddenly take a step back and realize what would make the code they've just worked on a lot better. But they are not yet experienced enough to write the code like that to begin with. Whereas veterans get most of the design right from scratch.

When learning programming, there's this process where you will always look at older projects you've made with dismay, realizing you would have done things very differently today. Over time, this dismay effect will fade as you get more experienced. When you eventually find yourself looking back at project you made a few years back thinking "hey this code is pretty good", then you probably have some 10+ years of full time programming experience.

(Not to be confused with programmers talking about "refactoring" the existing code written by others. You can be dead certain that they will eventually insist on reworking it at some extent, perhaps particularly so if they are veterans.)

---

You could also hand them a big code base and ask them to find out where it does one particular thing, or where a particular (planted) bug is located. Then see how they go about it.

Expected behavior would be lots of file content searching, following identifiers deeper and deeper, doing "divide and conquer" - all while you truly don't care about what anything does, not until you've dug down to the part you are interested in and only then stop and consider - what is this file, what does it do, what's the specifics of the code. If they have found the place you asked for within reasonable time, they are good enough. If they have also found the cause of the bug or have an idea about a solution, even better.

Whereas someone starting at reading project documentation and looking at big picture stuff like file hierarchies probably isn't very experienced - that's all well and good but there's a time and place for everything. During an interview or when putting out fires by trying to quickly patch a fatal bug, there's no time to stop and smell the flowers.

Answer 16

As a developer I am tired of seeing technical-tests where I had to find the ships in an M x N array for jobs where later you will have to build an API with CRUD's for different entities: no relationship between technical-test and the real final position.

For me, the main problem with interviews is that they are focused in "do you know how to solve this in-an-academical-way?" but later you won't have the same kind of problems in the real life, where you probably were more focused on architectural problems, or refactoring problems, or more high-level issues.

Most of the time the final algorithm is not the key and, more or less, anyone is able to solve the way for doing something at low-level (if/foreach/while/... there are few control structures) but not all the people is able TO THINK PREVIOUSLY about the problem itself and the best way for solving it, so I think the technical tests should be more focused on the way a candidate could face to problems instead of "this guy has forgot finishing the line with ';'"

When I had to hire engineers I focused on abstract concepts like "this person could face any problem and solve it", "this person could learn about our business in not so much time", "this person es naturally curious", "this person knows how to approach to the problem" or "this person has different ways for trying to solve problems" instead of "this person could solve NOW the problems we have", so I think you have to lead the conversation with the candidate to be able to answer this concepts and not to try to know if the candidate could write code fluently (if it's senior I assume he could), but the abstract concepts I have told could be potentially candidate weakness if he/she doesn't have it.

Answer 17

I'm actually going to announce unwelcome news, which is that there is no formula that discerns what the performance of a new developer will be when introduced to an arbitrary new organisation and new codebase.

Moreover, when a developer is found not to fit, there is no formula for discerning when the problem is the developer, or when it is the organisation or the codebase.

There's a temptation to think that different software developers are essentially in a relatively standardised and consistent occupation, and that the only variable is their talent or individual competency.

In reality, the roles and responsibilities of software developers are extremely various. It differs between firms, between firms of different sizes, between firms in different sectors, between teams in the same firm, amongst different members of the same team, and then everything I've just mentioned can vary significantly over time too.

This means that there is often a very modest correspondence between past experience and the proposed new role, not just in terms of technical skills, but in terms of relative organisational power and responsibilities.

And there is no widely accepted taxonomy of what all the different variables are on both sides, how they do or don't fit together or, crucially for you, how they would be measured except by trial and error.

Hiring managers may develop their own finer rules of thumb with experience, but these are often either tailored to the implicit context of their organisation (which are difficult to articulate), or they are unsubstantiated superstitions. For this reason, it's rarely useful to solicit specific selection rules from experienced hiring managers, especially from those not steeped in the context of your organisation - because you'll get a mixed bag of statements whose applicability you cannot discern.

The quality and consistency of codebases can vary too. The topic of software quality could fill a tome of commentary in its own right, but what everyone already knows is that there is some kind of relationship between the developer and the codebase, and that the codebase can be in such a condition as to cause misery to the developer and impede the progress of their work.

The most universal cause of codebases to which it is difficult to introduce new developers, is when the complexity of the accumulated code is not matched by the presence of any staff who are able to explain its workings - "able" in the sense that these staff are present in the organisation (often not), that they have time to talk and write, and that what they talk and write is sufficiently skilful and coherent.

My conclusion is not to indulge the fantasy that you can circumvent this suck-it-and-see process, and be prepared to acknowledge that the issues are with your organisation or codebase, or with the fit between these and a particular new developer, not necessarily with the developer intrinsically.

Answer 18

My two cents:

I'm a senior developer in a company that has a codebase spanning thousands of repos and trillions of lines of code. I handle it well, was not an easy process and didn't happen overnight.

Heres the thing:

If I was interviewing and the interviewer started asking silly technical questions or asked me to write code to flip a binary tree or something like that I'd end the interview and recommend they hire a graduate who remembers all that from their uni days.

You need to change your mindset of what you're asking senior developers.

Answer 19

Allow me to base my answer on a few facts (as mentioned by the OP) -

1. The candidate is good (technically).
2. They couldn't grasp the complete picture (from a domain/functionality perspective, I suppose).
3. They underperformed as per your expectation.

Now lets drill deeper with a few questions -

1. What was the experience level of this professional?
2. How large a project/product are we talking about?
3. Had you given enough walkthrough on the project/product?
4. What dev process are we talking about (Agile/waterfall...)?
5. What is the experience level (both total as well as on this particular 'app') of the team, in general?
6. What are your team coordination strategies (brownbag/brain-storming et al sessions)?

Remember: The solution was always there, even before the problem. If you didn't see it coming then now is the time to introspect.

Answer 20

For what it's worth, the best ways I can think of to directly test someone's ability to work with large systems - and I don't claim it's a good answer - would be to hand them a large code base and ask them to find the point at which a particular behavior is implemented (tests ability to reverse - engineer their way into existing solutions), or ask them to draw up proposed architecture of a solution rather than a full implementation of it, perhaps implementing part and stubbing out the rest.

Either would be a rather big ask for an interviewee - that's closer to undergraduate term project than nightly homework - so I suspect it's more common to just make the best guess you can based on their resume and ability to discuss how they would tackle architecture design and analysis, provisionally hire someone, have them dive in on real code, and then let them go if they drown during onboarding.

Hiring higher skills is more difficult, and more expensive, and more risky then hiring code monkeys.