4

Some background: I lead a small team of systems administrators that supports two different server rooms. Over the last couple of weeks, there have been problems with the power supplying the building such that we have been called on to safely power down the rooms on very short notice.

Yesterday in our management meeting, one of our managers stated to me his intention to develop a phone list of our team and post it so that everyone will know who to call when they have similar problems in the future.

My response was to ask if he intended to set up some sort of formal "on-call" arrangement (right now there is none) to provide this level of 24/7 support. He waved his hands and said that we need to just expect that salaried employees need to do what's needed to keep the system going. We are in an at-will U.S. state, so there is no employment contract in place that would clarify this.

Some of our systems are on a small UPS system designed only to handle brownouts or short (10-15 minute) outages, but not long-term outages. Because of this, what he is asking (even if he is not explicitly aware of it) is that someone on our team always be within 10-15 minutes of our facility.

To complicate matters, the building is secured after-hours by an alarm system that our group does not have access to, which means that our 15 minute response window must also include the response time from our security team.

A clean shutdown of all systems for one person can take more than 30 minutes to complete. More than one person working together can be somewhat quicker.

I feel that what the management team is asking for is unreasonable. I am concerned about the lack of additional consideration (i.e., money or comp time) for the inconvenience of being on-call, but also that if we agreed to it there are limitations outside of our control that would make it unlikely to meet his expectations. It seems we would be agreeing to be responsible for the inevitable failure that is weeks, months, or years away.

  • 1
    What is being done to investigate the on-going power failures? In other words, is this on-call commitment only a short term measure until the power is properly fixed? – Snow Nov 1 '17 at 14:51
  • A problem with the power lines to the building has been found and repaired which should take care of this month's panics, but the larger issue of responding to area-wide power failures, HVAC problems, etc. remains. – spuck Nov 1 '17 at 14:53
  • @user897243: To me, as someone who lives in an at-will work state, this isn't about worker rights but rather about expectations of the job being performed. Obviously the other managers believe his group has responsibility for this problem - and I agree with them. – NotMe Nov 1 '17 at 15:06
  • 1
    Why aren't the systems shutting themselves down in this situation? Shutdown should be automated or accessible remotely with an automatic notification to the admins. – 17 of 26 Nov 1 '17 at 15:34
  • @NotMe: I agree, our job is to keep the systems running. My biggest concern is that given the requirements and current constraints, doing so is impossible. – spuck Nov 1 '17 at 15:34
17

It's been my experience that network/system admins for 24x7 operations are always "on call". It's part of the job. The existing compensation package is almost always based on this.

As you are the leader (manager?) of the systems administrators I would have expected you to take charge on how to handle the ongoing power problems. After all these directly impact your ability to keep the systems up and running. It sounds like the other managers don't have faith that this is happening.

If I were in your shoes I would do the following (in no particular order):

  • Investigate the power problems. Determine if this is going to be an ongoing issue or if it's temporary. If it's temporary, for how long do you expect it to last. Make sure this information is communicated broadly and keep tabs on it.

  • If it's ongoing, then put together a plan to either move the data center to somewhere with better power or research other power options that would meet the business needs. For example, fuel based backup generators or even moving everything to be hosted by AWS. Sys Admins should be on top of things like this just like they handle backups.

  • You should also document the exact steps needed to take down, and restart, the operation. These steps should be both easily accessible and able to be handled by whoever is on call.

  • Take charge of putting together a rotating On Call list with a single phone number for others to call in case of issue. The common way of handling it is to acquire a company cell phone that gets passed around. The point here is to remove any question of who to call from the non-sys admins.

  • Once you have these things in place, discuss a change to compensation for those that are going to be in the On Call rotation. Maybe it's in extra $2,000/year; maybe it's an extra couple of vacation days. This shouldn't be tied to whether they actually receive calls or not, just tied to whether they actually fulfill the requirements of being On Call. Regardless, each and every call should be logged along with resolution and time spent so you have the needed information to reevaluate compensation later.

In other words, take charge before someone else forces a solution on you.

  • 2
    To me, this is the right answer. Take responsibility yourself or live with somebody else´s decision. Apart from that you could also improve the technical side. Servers should be able to message tec. team and shut themselves down safely before power runs out, automatically. – Daniel Nov 1 '17 at 15:06
  • @Daniel: I agree with server messaging. A well handled data center should be able to alert the admin team whenever a fault is detected - which includes power, heat, etc. This would obviate the need of others in the business letting them know when a problem has occurred. – NotMe Nov 1 '17 at 15:10
  • Thank you for the insightful comments. I left out some details to try to be concise and that I thought were distractions from the base question: 1. The server rooms are self-contained (no outside network connections) and moving our data off-site is not a possibility. We have shutdown/startup procedures documented. The current power issues are temporary, but we are at the mercy of our commercial power provider. – spuck Nov 1 '17 at 15:26
  • @spuck: Interesting problem. Definitely research the costs involved in getting an onsite backup generator to keep operations going. Management may not like the costs but they might actually be cheaper than dealing with downtime. – NotMe Nov 1 '17 at 15:39
  • 1
    Even if the systems don't have outside network access, they should be capable of detecting a problem and shut themselves down. – 17 of 26 Nov 1 '17 at 15:41
2

What your management team currently doing is short sighted and ultimately mis-guided in the long run. However, in the short run, their expectations appear to be reasonable - to have enough staff on hand to be able to timely service / troubleshoot the IT systems.

I would recommend you follow along in with your management team in the short term on an on-call basis for the sake of system stability and availability. Letting the systems go down entirely would look worse than the situation your are in right now, as it strengthens the belief that the system administration team is not able to support the business needs.

You have several very valid concerns in your post:

  • No UPS for all systems - what happens if the commercial utility company goes out of business or suffers a disruption? What happens to the data on that system your team supports?

  • HVAC issues - What happens if the data center suddenly overheats?

  • Single point of failure of a critical vendor - if the sole utility company were to suffer a disruption in its operations, your company is out of luck

Investment decisions are often driven by value added vs. the risk of not investing. To help obtain management buy-in in getting what your team wants - more resources and perhaps better systems, you can perform an business impact analysis (BIA) to determine the risk of not investing in additional resources or your team Your question did not provide key information to be included in such analysis such as the below: (not exhaustive)

After such an analysis, you can meet with the management team and present to them the impact and cost of not having adequate resources to maintain system uptime. An example of what you can say:

If [System X] were to go down, we stand to lose approximately $X in direct costs, such as revenue from customers, and $Y in Indirect costs, such as future income foregone due to damaged reputation and trust.

When costs are quantified, it becomes much harder for rational management to ignore, as its easier to recognize that not investing hurts the company in the long run - an irrational business decision.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.