How would you standardize on-call expectations across software teams when they've diverged at the same salary level?

Question

I manage a couple of software teams for a fintech SaaS product. We generally expect teams to maintain an on-call rotation among the devs so that there's someone to contact in case of an off hours emergency.

I recently found out from the manager for one of the teams that she's the only one on call in case of issues with her systems (of which there haven't been any in the year and a half I've been with the company, but it's a mission critical system). Apparently this was the standard put in place by the previous manager, and the devs on this team were hired with the expectation that they wouldn't have any on call duties.

I'd like this team to add a rotation so that this manager isn't on call 24/7 and to align the team with the others. However, I'm going to have a hard time changing their compensation if they ask for that in exchange for taking call, as they're in the same salary bands as all the devs on other teams that had different expectations set when they joined, and have been on call since they started with the company.

How would you navigate this?

Answer 1

I would like to go with a little frame challenge here, give you a different perspective you may have overlooked:

You have one of your mission critical teams that did not have a single incident during off-hours for 18 months!!!

How about you give every single one of them a big fat bonus? That is what pops into my mind. The last thing you should do with that team, is change something.

Assuming you don't live in a disaster zone where servers might get swallowed by an active volcano, software does not erode. It does not get old, it runs and runs and runs. You only need to contact the on-call person if said software is badly written, has bugs or fails through some other unforseen means.

Your team has managed to build a product that has not failed in a year and a half. That is amazing!

When I worked for big corporations with multiple teams, it were always the on-call members of the teams that got big credit. How their effort saved the corporation from paying hundreds of thousands in contractual penalities for the downtime avoided. How they slaved away on Saturdays and Sundays for the good of the company. How heroic their efforts and how great their spirit.

Yeah. Nobody asked why their crap even failed in the first place. Why that piece of bug ridden manure was even deployed untested on a Friday afternoon. Did not matter. Those who fixed it were heroes. It was like we had Firefighters that were really great at putting out kitchen fires and won tons of medals for it, because every time they left for work, they left the stove on.

Look at your teams. Look for the team that does not have incidents. That tests all their software. Deploys on mornings during the week so everybody is available should it fail despite the extensive testing. That are the heroes. Not the people who are so busy making their own life miserable and putting out the fires they themselves started. And probably make overtime pay on top while being incompetent.

So, a bit of advice regarding your team? You don't seem to have an actual problem. Don't artificially make one up for you to fix. It seems to be a great team. Leave it be.

And think about that bonus.

Answer 2

I worked at a place where they tried to add on-call responsibilities without any compensation. It didn't go well.

First they tried giving comp time, but nobody could take it because they still expected the same output during the week.

They dictated rules regarding how quickly we had to respond, but didn't want to pay for the hours where we would have to be on call. The programmers just ignored the emails and texts and waited until Monday to see what was urgent.

They had to drop the plan. They decided it was better to construct a staff just to be available 24/7.

but it's a mission critical system

That costs money to staff. If you can't wait until the next business day, then you have to compensate them for the time. If you have enough turnover you can add languages to the position descriptions for new employees. If that doesn't get the coverage you need, then you may need to give them a bump in pay.

Answer 3

Apparently this was the standard put in place by the previous manager, and the devs on this team were hired with the expectation that they wouldn't have any on call duties.

The developer have been hired for a role that doesn't require them to be on call, and their contracts will reflect this. If you want to change their contracts to ask them to do something more (i.e, be on call), then you need to offer something more in return (i.e, more money). What other people are earning, or what other people's contracts say is irrelevant.

First you need to define an SLA. How long do they have to answer a call? How long to be working on a problem remotely? How long to be in the office?

On call compensation typically has two components. The first is that you need to pay them a flat rate for all the time they're on call, regardless of whether or not they do work. This is because being on call significantly restricts their life (depending on the SLA). They can't drink, they can't travel anywhere without their laptop/phone/Internet, and they can't commit to doing anything that takes longer than you SLA.

The second component is for when they get a call, which should have a minimum time block. Even if you deal with a phone call in five minutes, if you got that call at 3AM then it's a significant disruption.

You also need to think about how you're going to schedule on call (no one is going to want to do it over Christmas), and how you cover people who've been called (if someone receives a 3AM call, you can't expect them to be in the office from 9-5 the next day).

Once you've worked out the terms that you want for the contract, you need to present it to them and see if they accept the changes and want to be on call. And if not, you either need to offer more money until they do (note that some people won't accept it no matter what you offer), or you need to find another plan.

You don't mention where you're based, but countries with decent employee rights don't just allow you to change try and change a contract and fire people who refuse to - so think carefully about your next steps if they decline the changes.

Doing on call properly is expensive. And it may be that when you look at the real cost, suddenly 24/7 isn't that important to the company any more.

Answer 4

I'm coming at this from the position of a software developer who's previously joined a large organisation in the UK with no on-call expectation initially, only to be told a year down the line that we were doing it (and being given no option).

To echo some of the other replies here; it doesn't go down well for anyone involved.

I can only speak from a UK perspective here. You should ensure that the employment contract mentions this. If it doesn't, you're going to have a hard time. This was my case: I did not go on-call – I dug in my heels, and the management were too inept to work with HR to change my contract.

Another thing to note is to be careful of your top talent. They will either leave for another company with no on-call expectation or simply be stubborn and difficult. It's a candidate’s market at the end of the day. In the end, I left the company mentioned above and started my own business. I consider that role the worst of my career - I also make sure to let friends and ex-colleagues know this, so they know to steer clear of that organisation.

As mentioned by some of the other answers here - the employees that are not stubborn and difficult may well just pretend they're available - internet connections at home can mysteriously "go down" over a weekend, as can mobile phone signals. So unless you want to pay through the nose for people's internet access and play all those games; re-think your approach.

If it's so important to you, use carrot, not stick. Involve the employee, and offer at bare minimum an amount for being on-call, and then an hourly rate if they are called out.

Being on-call - even if you're not actually called out - is a huge inconvenience and interrupts people's down-time, which is essential for someone to come back Monday morning and do a good job for your company. Depending on the SLA, you cannot make plans and/or socialise, or simply just switch off and relax.

If attempt to take this away from people - expect a fight.

Answer 5

One thing to think about is that there is a difference between the appearance of being on call (such as an absence of announced objections to the imposition of such a responsibility), and really being available for calls when they happen.

The manager who is nominally available 24/7/365 is probably not reorganising her life to be always on-call.

In reality, she is ensuring the software (a "mission-critical fintech" application) meets a standard that never requires urgent out-of-hours attention.

Beyond that, her phone is on for all sorts of reasons (including personnel rather than software issues), and in a true once-in-a-career emergency with the software you'll have a chance that she's immediately available for work (and not heavily drunk, for example).

Similarly, on a team of a few people, there's bound to be always someone who can be got in the middle of the night for triple-time and a couple of days off afterwards. Such a policy need not be announced in advance, and the cost in wages would be trifling in the scheme of things.

Now, if you try to impose a new on-call rota in which people are specifically expected to be available for work out-of-hours, and do it without any genuine consent, two things will happen. These things will happen even if you impose it on new-starters, but they don't feel the responsibility is commensurate with their position and remuneration.

Firstly, you'll create a strong sense of grievance and upset staff relations, that could lead to departures, administrative burdens, and a loss of flexibility and discretionary labour in other areas of work.

Secondly, beyond those who challenge the imposition in an overt way, many more will either just ignore the responsibility (so that the appearance of being on-call is really just a charade, and they go about their lives out of work as if not on-call), or may even spitefully ignore calls when they happen.

So I'd think about how else the problem can be tackled, or whether there even is a problem in the status quo.

Answer 6

That ship has sailed, forget about it.

I've been in the developer situation, largely. Admin job, company needed on-call duty when we had been hired without. CTO was smart enough to understand he can't dictate it and decided to talk about it, explaining the situation and flat-out asked how much extra we want.

We asked for a day to decide, did some research, thought about the impact it would have on our lives, then asked for 30% plus a few stock options. We got it. (in our case it wasn't rotation, it was all of us, all the time - it was a small team)

And we were all young, without kids. If you have people working on that team who have a family, they might not agree for any amount of money.

On-call duty is a massive change to the private life of people. It's not something you handle the way you seem to look at it. It's essentially on the same level as asking people to do a completely different job than the one they've been hired for. It's invasive enough that people will quit if you try to force it on them, especially in the job market today.

Sorry man. Elvis has left the building. You can stop screaming.

Answer 7

A perspective that is somewhere in between most of the current answers:

If you can afford to do this gradually (and with no failures in 18 months, it sounds like you probably can), blend this with other aspects of the job that are tied to salary. You've actually already got exactly that situation, and it will make the pill easier to swallow.

What do I mean by this? Simply that there is, in every job I've worked so far, an expectation (often unspoken, which is not good, but there none the less) that for both "technical" track folks and their direct management, the more senior you are, the higher the chance that you will get pulled into an emergency, whether or not you are officially on call.

So if you can get by with setting a "phased in" expectation that as people rise in seniority, they will (slowly) get more on-call duties, usually under very rare circumstances, and you are willing to accept that they do what your manager is already doing — to wit, not being "traditionally" on call, but just being willing to pick up the phone if work calls on a weekend — then the low failure rate may mean that you can, realistically, accept the risk of being down long enough for someone to sober up if they need to, or get back from the movie, or whatever they were doing. This may well simply be a matter of one of your most senior people being willing to take on what they currently perceive as "management responsibility" (after all, your manager is doing this right now, and honestly, I've never known a good manager who wasn't this way) as part of career growth.

Conversely, if what you need is a guarantee of someone being available to fix the problem, no matter when it happens… actually, you don't, and you can't get it anyway. There's always the possibility an asteroid / tsunami / societal collapse / whatever will wipe out your entire team at once anyway. Of course, normally that gets ignored under the logic "you're going to have bigger things to worry about, in that case". And that's fair, but it highlights the fact that this whole scenario is about risk mitigation. So instead, look at what the costs of an outage are, what your options are for dealing with mitigating the impacts, and what those costs look like, and find a balance.

Remember that from the employee's perspective, they don't care (and very likely don't know!) what "others like them" are making. Companies put a non-trivial amount of effort into trying to make sure this stays true, in fact. So to them, what some other team is making for doing some "similar" job that includes on-call pay means nothing whatsoever. Their job is the one you're proposing change, in a way that will, at best, be inconvenient and mildly problematic for them, and at worst could be wildly disruptive. And you're seriously thinking you should ask them to do this without any form of compensation, and expecting you won't lose 80% or more of them, in particular all the ones who are the "best" (because they will have the easiest time leaving)? If you really want to replace the entire team, there are far cleaner ways to go about it.

We had a situation a while back at my current employer where, due to various miscommunications, an SLA expectation was set of a group that wasn't prepared for it, more or less out of the blue. The only saving graces — and they were barely enough to keep half of the team members from sending out resumes that same day — were that:

• It was a very temporary arrangement (a couple of weeks)
• Management openly admitted that they fouled up (not "mistakes were made", but rather "I made a mistake")
• It was presented as a request that the team "come together to help weather this", much like one would with, for example, an "all hands on deck" crisis
• The team already had a strong (and actively encouraged) sense of camaraderie, such that folks trusted each other to be able to 'cover' if someone truly wasn't able to be on call, etc.
• The team was extensively reassured by the folks they would escalate problems to (myself and my boss) that while they might individually be on call on a rota, we would be available for escalations 24 x 7 — and they had several years of proof of this being true already.

That last part is more crucial than it might seem. If there is one thing worse than being "on call when you didn't negotiate for that up front", it is being in that situation and not having anyone to fall back on who is better at problem solving than you are, if you truly need it. I don't have that, but I'm also compensated for that aspect of my job — and it doesn't generally stress me out as much as most people seem to get.

But honestly? We have 24x7 operations support, "mostly 9-5" core dev support, and a few of the more senior devs are okay with getting pulled in ad-hoc — and that has proven to be good enough to support several thousand installations across the US. If you need more reliable availability than that… you really shouldn't be flinching at paying for it. If it wasn't negotiated up front, treat it as the cost of a mistake, make sure your hiring guidelines reflect what you actually need, and move on.

Answer 8

What wasn't mentioned at all: Some people just don't like having to interact with customers. For ages, I had managers who made more money than I did, but they all had to do the (to me) sh***y jobs that I wouldn't have wanted to do, so I was quite happy with that.

So it's not only men and women with family who will refuse to go on call, but also people who just don't like that kind of job, plus of course everyone who may be willing to do it, but not for free. And right now, if you change a software job in a way that the employee doesn't like, they will quickly become someone else's employee.

Answer 9

Do you even need someone on call?

This system has had no issues in 18 months. It is apparently a mature and entirely stable system with no issues. Does it in fact merit having someone specifically on-call to deal with emergencies?

Could you set up a more ad-hoc policy where if there's an issue out-of-hours you are able to call up your devs and compensate them heavily for their time?
In my experience most devs are.. Not necessarily happy, but understanding and prepared to deal with emergencies as they arise.

Then you aren't relying on a single individual person anymore, nobody will be made upset by being required by contract to be available out-of-hours, and everyone wins.
Most likely it's a policy that will not be tested often if ever, so it serves the company well to not be too doctrinally heavy-handed with it.

This is all predicated on the idea that you have multiple developers who have the knowhow to actually deal with issues with this system.

The reality is that stable-product aside, you've got a major Bus-factor risk here. If this developer quits, or is hospitalised, or on holiday, you have no recourse but to ask one of your other developers to step in anyway.
So make that your official approach.

Answer 10

How would you navigate this?

Generally you don't fix something that isn't broken. So I'd first rethink the need to go down this path.

If it was decided to be necessary I'd talk to the person who knows the team best which is their manager and discuss implementation.

devs on other teams that had different expectations set when they joined

Inertia and prior experience will make a change difficult so expect pushback. All you can do is make it mandatory and their manager will have to make it happen. This may mean that in fact the manager handles all the calls anyway. But in theory you have met your goal. And since they don't actually get any calls you'll never have to worry about it.

Answer 11

That team is apparently working correctly. You do have someone on-call 24/7, as you need. If that works for the team, I wouldn't try to change it, specially given that it's a "change for nothing".

Additionally, it's not bad that the manager is on-call as well, as that will provide the right incentive that the system must not fail outside working hours (nvoigt answer).

The point that concerns me is if she would be able to actually fix the mission critical system should it fail. There are managers that know their systems inside-out, and others that would be unable to do anything about it.

Even if your manager is one of the later, there's nothing wrong with having her as the official person on-call, and if that's outside her expertise, that she then calls someone else (perhaps choosing whom based on the type of failure). Maybe that team does have the expectation that on such failure they would be called (listed as being on-call or not). It's up to your manager to ensure that she would be able to get that support if needed, though. (Note this is not specific to this team, in the other team as well it may be desirable on a given crisis to get "more hands on deck", or asking someone that has more experience/recently worked in the broken subsystem).

You may want to perform some drills on your teams, to ensure they are really available in the stated SLA: phones are picked up at night, people on-call is not far from their laptop and so on.

Nevertheless, if those systems are really mission critical, you should probably an escalation path (on both teams) in case it was not possible to contact with the primary person on-call (which does not require any fault from the person on-call, they could fell ill during the weekend, the phone could break, the telephone network suffer a failure on their home area…).

Answer 12

I'd like this team to add a rotation so that this manager isn't on call 24/7 and to align the team with the others. However, I'm going to have a hard time changing their compensation if they ask for that in exchange for taking call, as they're in the same salary bands as all the devs on other teams that had different expectations set when they joined, and have been on call since they started with the company.

How would you navigate this?

I would standardize the on-call rotation across all the teams you manage.

I wouldn't change their salary because of this. As you have indicated, their salary bands match those of the other teams who are on-call, so they haven't given up salary. Instead, I would offer comp time to everyone on all teams if they had to spend significant time handling an issue when it was their turn in the rotation.

That's the way it worked every place where I was employed, including fintech SaaS product companies. Everyone was treated the same. It seemed fair.

As the manager, you should also be working at the same time to minimize the need for handling issues in the first place. All such issues should be logged and analyzed to determine why it happened, and what could be done to prevent it from happening in the future. If you see a pattern in these issues, that's an obvious target for system and/or process improvement.