How can I convince management that their on-call expectation is unreasonable? Am I unreasonable?

Question

Some background: I lead a small team of systems administrators that supports two different server rooms. Over the last couple of weeks, there have been problems with the power supplying the building such that we have been called on to safely power down the rooms on very short notice.

Yesterday in our management meeting, one of our managers stated to me his intention to develop a phone list of our team and post it so that everyone will know who to call when they have similar problems in the future.

My response was to ask if he intended to set up some sort of formal "on-call" arrangement (right now there is none) to provide this level of 24/7 support. He waved his hands and said that we need to just expect that salaried employees need to do what's needed to keep the system going. We are in an at-will U.S. state, so there is no employment contract in place that would clarify this.

Some of our systems are on a small UPS system designed only to handle brownouts or short (10-15 minute) outages, but not long-term outages. Because of this, what he is asking (even if he is not explicitly aware of it) is that someone on our team always be within 10-15 minutes of our facility.

To complicate matters, the building is secured after-hours by an alarm system that our group does not have access to, which means that our 15 minute response window must also include the response time from our security team.

A clean shutdown of all systems for one person can take more than 30 minutes to complete. More than one person working together can be somewhat quicker.

I feel that what the management team is asking for is unreasonable. I am concerned about the lack of additional consideration (i.e., money or comp time) for the inconvenience of being on-call, but also that if we agreed to it there are limitations outside of our control that would make it unlikely to meet his expectations. It seems we would be agreeing to be responsible for the inevitable failure that is weeks, months, or years away.

Answer 1

It's been my experience that network/system admins for 24x7 operations are always "on call". It's part of the job. The existing compensation package is almost always based on this.

As you are the leader (manager?) of the systems administrators I would have expected you to take charge on how to handle the ongoing power problems. After all these directly impact your ability to keep the systems up and running. It sounds like the other managers don't have faith that this is happening.

If I were in your shoes I would do the following (in no particular order):

• Investigate the power problems. Determine if this is going to be an ongoing issue or if it's temporary. If it's temporary, for how long do you expect it to last. Make sure this information is communicated broadly and keep tabs on it.

• If it's ongoing, then put together a plan to either move the data center to somewhere with better power or research other power options that would meet the business needs. For example, fuel based backup generators or even moving everything to be hosted by AWS. Sys Admins should be on top of things like this just like they handle backups.

• You should also document the exact steps needed to take down, and restart, the operation. These steps should be both easily accessible and able to be handled by whoever is on call.

• Take charge of putting together a rotating On Call list with a single phone number for others to call in case of issue. The common way of handling it is to acquire a company cell phone that gets passed around. The point here is to remove any question of who to call from the non-sys admins.

• Once you have these things in place, discuss a change to compensation for those that are going to be in the On Call rotation. Maybe it's in extra $2,000/year; maybe it's an extra couple of vacation days. This shouldn't be tied to whether they actually receive calls or not, just tied to whether they actually fulfill the requirements of being On Call. Regardless, each and every call should be logged along with resolution and time spent so you have the needed information to reevaluate compensation later.

In other words, take charge before someone else forces a solution on you.

Answer 2

What your management team currently doing is short sighted and ultimately mis-guided in the long run. However, in the short run, their expectations appear to be reasonable - to have enough staff on hand to be able to timely service / troubleshoot the IT systems.

I would recommend you follow along in with your management team in the short term on an on-call basis for the sake of system stability and availability. Letting the systems go down entirely would look worse than the situation your are in right now, as it strengthens the belief that the system administration team is not able to support the business needs.

You have several very valid concerns in your post:

• No UPS for all systems - what happens if the commercial utility company goes out of business or suffers a disruption? What happens to the data on that system your team supports?

• HVAC issues - What happens if the data center suddenly overheats?

• Single point of failure of a critical vendor - if the sole utility company were to suffer a disruption in its operations, your company is out of luck

Investment decisions are often driven by value added vs. the risk of not investing. To help obtain management buy-in in getting what your team wants - more resources and perhaps better systems, you can perform an business impact analysis (BIA) to determine the risk of not investing in additional resources or your team Your question did not provide key information to be included in such analysis such as the below: (not exhaustive)

• Criticality of the IT systems to key business processes
• Opportunity cost of downtime (lost revenue, damaged company trust among customers...etc.)
• Agreed upon recovery point objective and recovery time objectives

After such an analysis, you can meet with the management team and present to them the impact and cost of not having adequate resources to maintain system uptime. An example of what you can say:

If [System X] were to go down, we stand to lose approximately $X in direct costs, such as revenue from customers, and $Y in Indirect costs, such as future income foregone due to damaged reputation and trust.

When costs are quantified, it becomes much harder for rational management to ignore, as its easier to recognize that not investing hurts the company in the long run - an irrational business decision.