6

I am working in a quite big IT company in Europe. Recently we faced a series of structural changes, and one of them was "join the modern world-wide trend" and distribute on-call duties among the teams. I can understand the business motivation to reduce costs, but it looks like the employees are not so enthusiastic. Some developers are not wanting to learn the "new and exciting world of devops", some just want to keep their existing work/life balance, some feel abused with the proposal to spend 5 nights and weekends under house-arrest (SLA 5 mins) for approximately half-day salary. Personally, makes me sick that a previously good company (which I signed for) disrespects its own employees in such a bad way.

Arguments from management circulated around:

  • every modern/cool startup is doing that
  • just write your code perfectly and you will never get the call
  • we will not compensate incidents separately, because team will start to put bugs intentionally into production (????)
  • most cases should be just acknowledged and fixed "first thing tomorrow"
  • this is fun, to learn tons of new things
  • you are getting compensated!

Should I say that all of these are poorly made manipulations, and almost all of team members understand this. We already started losing our most experienced/valuable developers. Actually I have some thoughts in my mind, that such bad play can be a result of invisible (for us) radical changes on the investor<>company level. For example: "cut the costs by 10% at any cost!".

One team is especially struggling because it is responsible for maintenance -- tons of legacy and buggy software. For fixing and updating which, as usual, there is no proper budget. Management doesn't want to trade on-call for extra days off, either.

So my questions are:

  • what is the best individual strategy for minimizing the effect of such changes?
  • what else can we (as a team) try to bargain better conditions?

UPDATE: in our contracts it states that "company can involve outside of working hours", but it was really rare cases before, and I was promised that "we don't have overtimes". According to the TOP: "new contract version will be prepared soon. Every new hire will be informed about on-call."

Improve this question
5
  • The short answer is: read your employment contract. If it says you can be asked to be 'on-call' then you don't have much of a leg to stand on. If it doesn't then it's unlikely that the company can unilaterally change the contract (otherwise what would contracts be worth?).
    – brhans
    Jun 26, 2017 at 21:43
  • 7
    In most European countries if your contract is a subject of labour law then you have strong protections against overtimes. There are limits to max hours per week you should work and any more time (and being available within 5 minutes is considered to be at employer disposal) falls under overtime rules which usually means higher pay or return in out-of-work hours. Also changing policy from occasional additional work to strict on-call routine could be considered a change of contract. You should talk to a lawyer for guidance regarding labour work rules.
    – Zefiryn
    Jun 26, 2017 at 21:58
  • I dont feel comfortable to escalate to the labor union things. Actually i never even heard about such things in our or any other company in the city here.
    – user1411968
    Jun 26, 2017 at 22:08
  • 5
    "we will not compensate incidents separately, because team will start put bugs intentionally to the prod" this points to a lack of trust, which in turn points at underlying issues. It might be worth looking into this further, because if your management doesn't trust you, you're going to be hosed either way.
    – Erik
    Jun 27, 2017 at 6:14
  • 1
    most cases should be just acknowledged and fixed "first thing tomorrow" Then it is sufficient for me if I get an email that I will effectively read first thing in the morning (maybe earlier, but that's my decision).
    – skymningen
    Jun 27, 2017 at 9:24

3 Answers 3

Reset to default
10

what the best individual strategy for minimizing effect of such changes ?

Well, if you are lucky, there is enough guys who see this as a good opportunity to grab more money. Let them have it. Don't volunteer for anything, they might just fill the on-call schedule without you. Don't underestimate greed as motivation.

You could request help. Somebody must have been on call before. Ask them how they managed. If you are truly agile and this is going to the teams as devops, ask why the ops aren't integrated into the dev teams then. They aren't going to be fired, are they? So go and learn from them or better win them over as new team members.

what else we (as a team) we can try to bargain better conditions ?

You pool your resources and make this a team effort. First of all, talk to a lawyer. Now I don't say "get a lawyer involved" as in have him talk to the company. Your company does not need to know there is a lawyer involved. But if you split the costs, you can have professional support for very little individual contribution. I say this because some things you mention definitely rub me the wrong way for a European working context. I cannot tell if they are illegal, because I have no idea which country you are in and I'm not a lawyer, but a 5 minute SLA is no longer "on-call". "On call" means you have spare time, with some reasonable constraints (like don't be drunk, don't be out sailing) and if someone calls, you will get to work. 5 minutes reaction time means you cannot by definition, do anything but watch your phone. 5 minutes? I have been in the restroom for longer. Walking my dogs takes longer. God knows what might happen if I actually want to eat or shower. A 5 minute response time is possible but it is not "on call". It is work time, to be compensated in full and in some European countries, is not even allowed after you already did your day job because that would be too many hours a day. "On call" does not fall under minimum wage laws... but normal work time does. So go see a lawyer, to be informed what is legal and what is not in your jurisdiction. They have a company lawyer, don't fight unarmed in this one. If you share the costs across the team, a lawyer is dirt cheap.

Improve this answer
2
  • I really like your comment. I have got root of the problem. I am not against such things in general and definitely many good companies already doing that. But:
    – user1411968
    Jun 27, 2017 at 9:54
  • we are not Amazon (Netflix, put cool name here). Some young engineers were enthusiastic about new experience / money, but now some months later they are realized that this is not rewarded properly, so looks like note enough "enthusiasts", so here we are coming to the next question, how to get better conditions or refuse to accept this burden in polite(safe) way
    – user1411968
    Jun 27, 2017 at 14:49
0

First of all, the sky isn't falling - as you note, this is increasingly becoming a common practice, and for good reason - letting a team write code and then give it to someone else to deal with the problems violates every Lean and Agile tenet of fast feedback. So you can just quit, but more and more companies are doing this. It's time to start eating your own dog food, as they say.

So on to the question - how you can minimize the impact of oncall and/or get better conditions?

  1. Talk to an experienced operations person. They don't "sit at home" waiting for an incident. You get a laptop and a cell you can tether, or any number of other technical solutions, either your own or an "on call" one you pass around in the team, and you can respond just fine even if you went out to a movie or whatever. I can acknowledge incidents using the smartphone app from whichever incident management tool I'm using (Pagerduty and VictorOps are common) and look at what's going on in the monitoring app. There's a lot of just plain coping mechanisms you don't know because you're new to it. All the devs and operations engineers at my workplace are on call, but they live their lives while doing it.

  2. (Ideally your manager) Track the oncall activity. How many, when, how long. What are the root causes? This is an opportunity to drive improvements. When I was managing a team that had inherited an awful oncall burden (200-ish pages a week), converting the team from whining to tracking had a big effect. Because all techies whine, and all managers are sick of hearing it. But if you can provide a spreadsheet showing "hey there's an average of 20 incidents generating 8 hours of off hours oncall activity a week!" or whatnot, they will be able to justify resourcing it more. Right now you're trying to argue against a what-if, which is never a strong position.

  3. When incidents happen, have blameless retrospectives and put work in tickets to fix the contributing causes, and to do whatever will make it easier to detect and remediate the issues. One of the primary reasons the management organization is making this change is to get the bugs fixed. You can't just sit there and suffer from bugs in the software you maintain and act helpless. You're not - drive improvements. This combined with #2 will help you get dev time dedicated to fixing bugs.

  4. Negotiate incident severity and SLAs. 5 minute response is definitely unreasonable, and unless there's people hooked up to the software you write using it to breathe or there's $5M/minute flowing through your trading app, it's overkill. But it sounds like they've already thought about severity and what things need immediate response vs can wait. Participate in that definition to get more breathing room, to the degree that it's business appropriate. Once you have some incident retrospectives under your belt you can do a time analysis and show how busting hard on 5m response time isn't actually a meaningful factor in mean time to resolution.

It seems like you've already decided on the most negative possible response to this change. While there are certainly managers out there in the wide world somewhere that just want to stick it to you, in general tech managers, like everyone else, want to do right by their teams and do their job well. Don't assume malice.

You're welcome to I guess, but I've worked at many places who have pervasive oncall and/or have implemented it while I was there, and in general the fear of the change ahead of time is much worse than the actual situation once it's in play. I've only seen a couple people that left/were asked to leave as a result of developer oncall, and it was never because they had an objectively heavy burden from it.

Improve this answer
4
  • 2
    If the company has decent test procedures in place there should almost never be a problem. With tester buggy code shouldn't be going out to customers.
    – Snowlockk
    Jun 27, 2017 at 8:30
  • Well right. It only becomes a problem when the dev-test-release cycle is releasing junk, and is an effective solution to that - making the team actually own their product and be directly incentivised to fix the bugs and let fewer out.
    – mxyzplk
    Jun 27, 2017 at 12:57
  • 2
    testing - is another big problem here, because according to big-boss, qa - just waste of budget. And overall engineering culture (environments mgmt, releasing, testing) really far away from best practices. We have quite intensive turnover rate and problem of on-call is really not the biggest one. But on-call starts affect quality of live from now.
    – user1411968
    Jun 27, 2017 at 14:58
  • 1
    @mxyzpkl I have a problem with "making the team actually own their product" - well, they don't. The product does not belong to them. Value generated from it will not go to their pocket and if needed they can get fired. What they do own is their time outside of working hours. Taking what is not yours is not a proper solution for a flawed dev process, if the dev-tes-release cycle produce junk it means that the process is flawed. They need people around the clock fixing bugs? They should hire people to work night shifts. Only high ranking managers who own equity should be "on - call"
    – Yuval Perelman
    Jun 28, 2017 at 11:25
-5

I'd argue that as a team being "On-call" isn't a huge issue so long as responsibilities are spread out enough that you aren't on call more than 5-10% of the time and you are adequately compensated when there is a call. If you take a call outside of working hours you should be paid 1.5-3x of your hourly wage or hours for a minimum of 30 minutes should you get any call. Develop clear guidelines of what constitutes a "drop everything and solve this" sort of problem and what constitutes a "log the issue and deal with it first thing tomorrow".

While management doesn't want to trade for days off, I'd push along the lines of "under typical circumstances no time off for on call however more than 5 hours of off hours calls in a week requires a day off to match it". This would allow for management to get their way for the most part but would protect those that would have the worst impact from this policy.

As an individual, look for team mates to trade the time away with. Lots of people love on call as extra pay for very little extra work. Feel out teammates to see if they don't mind taking the week for the extra cash.

Improve this answer
6
  • 30 mins is far to low FYI in the UK BT's on call was a 4 hour minimum call out at x1.75 after 8 2x Saturday an Sunday 2x plus a day
    – Neuromancer
    Jun 26, 2017 at 21:58
  • 30 min is what we use at my site for calls that can be answered remotely and there is no shortage of people who are willing to take on call from those who don't want it.
    – Myles
    Jun 26, 2017 at 22:11
  • Without a high minimum there is no disincentive to reduce callouts which is what you should be doing
    – Neuromancer
    Jun 26, 2017 at 22:13
  • For calls that can't be answered remotely, our minimum is quite a bit higher (5 hours) and there strict rules about numbers of hours worked in a 24 hour period. Our organization may be a bit different when it comes to minimizing callout as it's a 24 hour chemical processing facility. Labor costs are a pittance compared to what happens if the plant goes down, so callout isn't even close to a driver when it comes to plant reliability.
    – Myles
    Jun 26, 2017 at 22:20
  • Being on call isn't a huge issue as long as you're appropriately compensated and knew about it when you joined the company.... but low on-call pay is a huge insult to existing staff: you lose a huge amount of freedom in your personal time, in exchange for very little compensation. If people volunteer, that's fine, but as an industry we need to get away from this assumption that it can be imposed by management.
    – Jon Story
    Jun 28, 2017 at 16:28

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .