A friend of mine is working in a company which is developing and maintaining a set of mission critical products. If one of those products fail at 3 AM, it is mandatory to do something immediately to get it back online, including solving the issue in the source code itself.
The company is now searching for a developer capable of solving issues related to the development itself. The friend will be in charge of the technical part of the interview, and asked me for hints.
Actually, a good candidate must be able to solve technical problems under severe stress. If the application stops working on the server of a customer, the company is paying a huge penalty per hour of downtime, meaning that the developer will know that while he is currently working on the issue, the company loses thousands of dollars. There may also be indirect pressure from angry customers¹.
How can I emulate or test such a stressful situation during an interview?
All the usual elements used during the interview to test how an interviewee would handle the stress² seem too small compared to what the person would do once hired, i.e. being required to rush in the middle of the night to the workplace in order to solve a bug as fast as possible.
Asking for previous similar experience seems harsh too. I'm not sure whether it is frequent to find workers who had worked in such frightening conditions.
So how can I gauge this skill during an interview? How can I know someone has this ability to work under stress before they're hired?
Update: more than two years passed since I asked this question. I kept regular contact with my friend, and was glad to hear a lot of good news. The company had a chance to hire some talented persons who took a bunch of good ideas and pushed them up to the management which was clever enough to accept them, by trusting those people unconditionally.
Among the changes, the development workflow was entirely redesigned, with the introduction of DevOps which works surprisingly well. Other measures related to code reviews, tests and the quality of code ensured that the product became progressively better. Now the developers are not afraid any longer to push their changes to production, because they are pretty sure those changes won't break anything.
There were difficult choices to do as well. Several programmers and managers were fired, because of their lack of skills and their unwillingness to learn, including a manager who were in this company for twenty years. Also were painful the decision to outsource a few jobs.
This being said, the company became much more mature, and, more importantly, much more pleasant place to work. Nobody screams at developers because of a regression they introduced, because there are none (at least not severe enough), and nobody should wake up at 3 AM because a problem was discovered and should be solved ASAP.
Surprisingly, the solution was very simple: the company now have three teams in three countries around the world, geographically chosen so that an evening in one country corresponds more or less to the morning in another country (there is, I believe, a three hours gap at some point that they have to manage by asking employees to start to work later than usual and to end later as well). This allows the permanent availability of skilled personnel whenever a problem occurs.
The only problem is that a bunch of services is still hosted in-house. This means that the company should have system administrators at reach day and night, on location, in case where a problem occurs to the data center itself (UPS failures, flood, servers starting to shut down unexpectedly, etc.)
This problem is about to become obsolete. The company is migrating its services to Amazon EC2, which appears to be slightly more expensive, but also much more reliable: a good price to pay.
I'm both happy and sad: happy to hear that the company is doing well, and sad telling myself that I focused my attention purely on the hiring process itself, instead of having a broader look at the problem, as any developer is expected to do. I had all the necessary info to think about a larger picture, but I haven't seen that the solution to hiring persons ready to wake up at 3 AM is to simply make the original problem irrelevant, would it be by outsourcing or by migrating to cloud computing.
This could be a good lesson—at least I consider it like one for me. Instead of looking for management-centered solutions to technical problems which often make employees unhappy, one should look at technical solutions to management problems. This case is a perfect illustration of that.
¹ Developers have a workspace separated from support department, but at night, I'm pretty sure that if an issue is found, there would be only two people in the company's building: the developer and the person from the support. While the support person will handle all the calls outside the developers workspace, the two will still talk together, so the developer would know that the customer is expecting a quick response.
² For example asking the interviewee to read or write code during a limited amount of time, or asking technical questions without telling if the answers are right or wrong, etc.