"Unexpected" on-call duties

Question

I am working in a quite big IT company in Europe. Recently we faced a series of structural changes, and one of them was "join the modern world-wide trend" and distribute on-call duties among the teams. I can understand the business motivation to reduce costs, but it looks like the employees are not so enthusiastic. Some developers are not wanting to learn the "new and exciting world of devops", some just want to keep their existing work/life balance, some feel abused with the proposal to spend 5 nights and weekends under house-arrest (SLA 5 mins) for approximately half-day salary. Personally, makes me sick that a previously good company (which I signed for) disrespects its own employees in such a bad way.

Arguments from management circulated around:

• every modern/cool startup is doing that
• just write your code perfectly and you will never get the call
• we will not compensate incidents separately, because team will start to put bugs intentionally into production (????)
• most cases should be just acknowledged and fixed "first thing tomorrow"
• this is fun, to learn tons of new things
• you are getting compensated!

Should I say that all of these are poorly made manipulations, and almost all of team members understand this. We already started losing our most experienced/valuable developers. Actually I have some thoughts in my mind, that such bad play can be a result of invisible (for us) radical changes on the investor<>company level. For example: "cut the costs by 10% at any cost!".

One team is especially struggling because it is responsible for maintenance -- tons of legacy and buggy software. For fixing and updating which, as usual, there is no proper budget. Management doesn't want to trade on-call for extra days off, either.

So my questions are:

• what is the best individual strategy for minimizing the effect of such changes?
• what else can we (as a team) try to bargain better conditions?

UPDATE: in our contracts it states that "company can involve outside of working hours", but it was really rare cases before, and I was promised that "we don't have overtimes". According to the TOP: "new contract version will be prepared soon. Every new hire will be informed about on-call."

Answer 1

what the best individual strategy for minimizing effect of such changes ?

Well, if you are lucky, there is enough guys who see this as a good opportunity to grab more money. Let them have it. Don't volunteer for anything, they might just fill the on-call schedule without you. Don't underestimate greed as motivation.

You could request help. Somebody must have been on call before. Ask them how they managed. If you are truly agile and this is going to the teams as devops, ask why the ops aren't integrated into the dev teams then. They aren't going to be fired, are they? So go and learn from them or better win them over as new team members.

what else we (as a team) we can try to bargain better conditions ?

You pool your resources and make this a team effort. First of all, talk to a lawyer. Now I don't say "get a lawyer involved" as in have him talk to the company. Your company does not need to know there is a lawyer involved. But if you split the costs, you can have professional support for very little individual contribution. I say this because some things you mention definitely rub me the wrong way for a European working context. I cannot tell if they are illegal, because I have no idea which country you are in and I'm not a lawyer, but a 5 minute SLA is no longer "on-call". "On call" means you have spare time, with some reasonable constraints (like don't be drunk, don't be out sailing) and if someone calls, you will get to work. 5 minutes reaction time means you cannot by definition, do anything but watch your phone. 5 minutes? I have been in the restroom for longer. Walking my dogs takes longer. God knows what might happen if I actually want to eat or shower. A 5 minute response time is possible but it is not "on call". It is work time, to be compensated in full and in some European countries, is not even allowed after you already did your day job because that would be too many hours a day. "On call" does not fall under minimum wage laws... but normal work time does. So go see a lawyer, to be informed what is legal and what is not in your jurisdiction. They have a company lawyer, don't fight unarmed in this one. If you share the costs across the team, a lawyer is dirt cheap.

Answer 2

First of all, the sky isn't falling - as you note, this is increasingly becoming a common practice, and for good reason - letting a team write code and then give it to someone else to deal with the problems violates every Lean and Agile tenet of fast feedback. So you can just quit, but more and more companies are doing this. It's time to start eating your own dog food, as they say.

So on to the question - how you can minimize the impact of oncall and/or get better conditions?

1. Talk to an experienced operations person. They don't "sit at home" waiting for an incident. You get a laptop and a cell you can tether, or any number of other technical solutions, either your own or an "on call" one you pass around in the team, and you can respond just fine even if you went out to a movie or whatever. I can acknowledge incidents using the smartphone app from whichever incident management tool I'm using (Pagerduty and VictorOps are common) and look at what's going on in the monitoring app. There's a lot of just plain coping mechanisms you don't know because you're new to it. All the devs and operations engineers at my workplace are on call, but they live their lives while doing it.

2. (Ideally your manager) Track the oncall activity. How many, when, how long. What are the root causes? This is an opportunity to drive improvements. When I was managing a team that had inherited an awful oncall burden (200-ish pages a week), converting the team from whining to tracking had a big effect. Because all techies whine, and all managers are sick of hearing it. But if you can provide a spreadsheet showing "hey there's an average of 20 incidents generating 8 hours of off hours oncall activity a week!" or whatnot, they will be able to justify resourcing it more. Right now you're trying to argue against a what-if, which is never a strong position.

3. When incidents happen, have blameless retrospectives and put work in tickets to fix the contributing causes, and to do whatever will make it easier to detect and remediate the issues. One of the primary reasons the management organization is making this change is to get the bugs fixed. You can't just sit there and suffer from bugs in the software you maintain and act helpless. You're not - drive improvements. This combined with #2 will help you get dev time dedicated to fixing bugs.

4. Negotiate incident severity and SLAs. 5 minute response is definitely unreasonable, and unless there's people hooked up to the software you write using it to breathe or there's $5M/minute flowing through your trading app, it's overkill. But it sounds like they've already thought about severity and what things need immediate response vs can wait. Participate in that definition to get more breathing room, to the degree that it's business appropriate. Once you have some incident retrospectives under your belt you can do a time analysis and show how busting hard on 5m response time isn't actually a meaningful factor in mean time to resolution.

It seems like you've already decided on the most negative possible response to this change. While there are certainly managers out there in the wide world somewhere that just want to stick it to you, in general tech managers, like everyone else, want to do right by their teams and do their job well. Don't assume malice.

You're welcome to I guess, but I've worked at many places who have pervasive oncall and/or have implemented it while I was there, and in general the fear of the change ahead of time is much worse than the actual situation once it's in play. I've only seen a couple people that left/were asked to leave as a result of developer oncall, and it was never because they had an objectively heavy burden from it.

Answer 3

I'd argue that as a team being "On-call" isn't a huge issue so long as responsibilities are spread out enough that you aren't on call more than 5-10% of the time and you are adequately compensated when there is a call. If you take a call outside of working hours you should be paid 1.5-3x of your hourly wage or hours for a minimum of 30 minutes should you get any call. Develop clear guidelines of what constitutes a "drop everything and solve this" sort of problem and what constitutes a "log the issue and deal with it first thing tomorrow".

While management doesn't want to trade for days off, I'd push along the lines of "under typical circumstances no time off for on call however more than 5 hours of off hours calls in a week requires a day off to match it". This would allow for management to get their way for the most part but would protect those that would have the worst impact from this policy.

As an individual, look for team mates to trade the time away with. Lots of people love on call as extra pay for very little extra work. Feel out teammates to see if they don't mind taking the week for the extra cash.