87

This is admittedly a mostly hypothetical question as I am not yet a manager, merely a technical lead, so if you don't want to waste time on a theoretical question, this is your warning.

Whether it be IT or pandemics, the incentives in life strongly skew towards slaying dragons rather than keeping them away.

The IT guy who keeps the system forever running is forgotten by management.People wonder why they have him around as "he doesn't do much." The IT guy who lets the system fail and then works 18 hours straight to fix it is considered a hero and rewarded.

In China, officials took a lot of steps to downplay the virus and hoped that it would go away for fear of it impacting their performance indicators.

Admittedly in my own career a couple years ago, I identified a bug one evening in a release and let it fail in production overnight instead of fixing it immediately as I wasn't going to stay late and have nobody know. We don't have an on call rotation as we assumed that things wouldn't fail in prod. I noticed it and went in early to fix it. The director of the department gave me a $2000 bonus for coming in so early to fix it. Had I stayed late to fix it, the company would have saved many thousands of dollars, but I would have gotten nothing.

Management at my current company couldn't be bothered to put together a work from home plan until last week when the city panickly ordered offices to close. Now we all use our personal laptops to remote in and access sensitive data as it never occurred to anyone that we should be prepared for that eventuality. I got hero points for showing people how to use remote desktop from their personal machines. Had I done that before the pandemic (I had no reason to as we shouldn't be accessing medical data on personal machines), I would have gotten no credit for it despite potentially saving two days of work time.

I am someone who aspires to management, but I cannot think of an incentive structure which encourages preparation over save the day heroics. I myself have a resume filled with save the day heroics.

I could see micromanagement being an option, where you demand a contingency plan for everything, but that seems very inefficient.

How could one reward preventing harm rather than mitigating existing harm?

Improve this question
1
  • Comments are not for extended discussion; this conversation has been moved to chat.
    – Neo
    Mar 26, 2020 at 19:21

8 Answers 8

Reset to default
49

My employer recently went through our first ISO-27001 information security certification. In order to pass we had to think through various failure modes and add procedures to work around them. We needed to test those procedures.

The failure modes were things like

  • office building becomes inaccessible due to disaster (fire, flood)
  • employee becomes hostile and tries to do damage
  • executive laptop stolen from car
  • cybercreep breaks in to production data center. etc etc

In the present disaster our office building has become almost inaccessible. So our ISO-inspired procedure for carrying on remotely turned out to be very useful indeed. I'm happy it was sitting there on a Google drive for us to follow.

Now, here's the thing. Our customers love love love that we have this ISO certification. It will pay for itself in increased sales in about a year. So, even if it's a bureaucratic pain in the ... neck, it's worthwhile.

To answer your question: If you want people to be proactive rather than reactive, standards-compliance is an excellent tool. You can point to the standard instead of saying "because I said so." And you can make heroes of the people who work on it via public recognition, bonuses, and so forth.

When I was wrangling ISO compliance, I had lots of conversations like this:

Employee: This is a waste of time and money, we'll never need it.

Me: Maybe not. But the ISO certification insists we work on it.

VP Sales: Hey, we need that ISO thing.

CEO: Getting ISO done by the end of the year is our priority.

Employee: OK, I'll work on it.

Me: How can I help?

We got it done, everybody involved got a bonus, and we had a blow-out holiday party to celebrate.

Improve this answer
6
  • 12
    This is why certain certification bodies have meaning. They're "worthless" until they're used. Emergency protocols are technically also worthless until said emergency occurs, then nobody dies instead of everybody dies.
    – Nelson
    Mar 26, 2020 at 15:40
  • 2
    The problem with certifications is that they will only account for expected problems. While unusable offices and destroyed hardware due to fire are expected (and hopefully mitigated by off-site backups), sending the whole company on home-office might not be. To be honest, 2 weeks ago on Friday, the scenario was "maybe one person needs to work from home to fulfill quarantine", on Monday it was the whole company going fully remote. That is a wholly different infrastructure you need.
    – Manziel
    Mar 26, 2020 at 15:57
  • 7
    @Manziel The reason to go for home-office may be unexpected, but your risk register simply has to say "office is unexpectedly closed for some reason". Could be fire, flood, murder, or alien invasion - it doesn't matter, so long as the servers are still running and connections are up.
    – Graham
    Mar 26, 2020 at 16:22
  • 11
    My employer attained ISO-27001 certification a few years ago, and since then, we have experienced: a hurricane that dumped 50+ inches of rain and paralyzed the city, making our HQ building inaccessible for days; a break in a 2.5m concrete water pipe that serves 1 million people, which shut down the city for a day or two; a plumbing issue local to our building that closed the building for a day and a half; and now the coronavirus stay-at-home order. We've maintained business continuity through all of this because we prepared and practiced. Everyone uses company-issued laptops and VPN.
    – shoover
    Mar 26, 2020 at 16:35
  • Also risk assessments. What is the risk of the office closing? What is the business impact if it does close (and you haven't prepared for it)? Granted, the current crisis is unprecedented in scope and duration, but a few weeks ago I was asked what the probability had been of the office closing, and my reply was that in my experience over the last four years, it had happened five times to different offices in the company (nearby terrorist incidents, nearby industrial incident, earthquake, total power loss). Of course, the probability goes up as the number of offices goes up...
    – Law29
    Mar 26, 2020 at 19:12
23

This is difficult.

Maybe eight hours work in time would have saved you 18 hours work later. But there are many potential problems that will take eight hours to fix in time and may cost 18 hours later. If you have ten such potential problems, but only one actually causes problems, you traded 80 hours of work against 18 hours heroic overtime later.

I think our IT got orders to prepare for everyone working from home, with highest priority, eight working days before the office closed down. And the last four days everyone carried their laptops from work to home and back every day, so everyone would be able to stay at home on a phone call without having to go to work once to pick up things. So in this case while the whole Covid pandemic was not foreseeable years ago, working from home was foreseeable two weeks ago.

Improve this answer
7
  • Maybe my management is just poorly informed, as they didn't prepare a plan even when schools closed.
    – pandemicincentives
    Mar 26, 2020 at 14:48
  • I've worked at plenty of places that don't use laptops, but rather desktops. Ordering enough laptops for the whole company, configuring them and transferring files, deploying them, and getting people to actually use them is unlikely to work for any company. Most companies can't afford that much expense all at once, no company has that much IT to handle that kind of workload, and if everyone is ordering laptops all at once, the supply chain can't handle that kind of load. You have to prepare for this long before the problems happen, not just react and hope you are accidentally prepared.
    – computercarguy
    Mar 26, 2020 at 17:06
  • @computercarguy This isn't a situation where remote employees need to be flexible and mobile. They can take their desktops home and leave them there for the next six months to a year.
    – StackOverthrow
    Mar 26, 2020 at 17:20
  • 1
    @MatthewGaiser, right, and how many problems is that causing your IT dept with security? How much of a problem is that with your legal dept for intellectual property? How much of a problem was it to set up all those different devices to access all the software you need for you job, or is it all just remote desktop? How slow is your access to the company network? You don't have to answer (and probably shouldn't), but these are some of the things that need to be considered before an emergency. Too many people think that all can be done easily, because they just don't know until its an emergency.
    – computercarguy
    Mar 26, 2020 at 19:35
  • 1
    @computercarguy I don't dare ask those questions... I do know there were many challenges.
    – Matthew Gaiser
    Mar 26, 2020 at 19:47
13

It seems to be human nature to reward heroes more than planners - so any cultural shift requires a quite massive change.

Rewarding the heroes must stop. At the very least, do not give bonuses for putting out a fire that their negligence essentially caused.

Instead, having post-mortem investigations after every incident could perhaps slowly change the culture to a more proactive one. After some time, you will realize who the proactive people are - as they will be the ones repeatedly invited to the post-mortems, from which the heroes will be absent.

Improve this answer
6
  • I think your last sentence is the wrong way around, i.e. the "heroes" will be the ones attending post-mortems, repeatedly.
    – Joe Stevens
    Mar 26, 2020 at 8:58
  • Agreed - some of them will show up, but they most often have nothing or very little to contribute, so management will need to step up and let the adults handle matters and not let heroes derail the process.
    – morsor
    Mar 26, 2020 at 9:16
  • I don't think there's anything wrong with rewarding heroics, and proper post-mortems are important (IME they always point to poor risk assessment) but the way to avoid problems is to do regular risk analysis, and act on it. Heroics tend to be done by individuals but good planning requires good management.
    – Robin Bennett
    Mar 26, 2020 at 14:55
  • 2
    @JoeStevens The "heroes" may be attending the postmortems as witnesses, but they won't (or shouldn't) be in control of the investigation as a whole. If a plane crashes but a "hero pilot" manages to save many lives, that doesn't mean you ask the pilot how to fix the problem that caused the crash. You ask the engineers who originally designed it.
    – alephzero
    Mar 26, 2020 at 15:15
  • 1
    When companies like Google publish bug-ridden garbage and take years to implement five-minute fixes for critical bugs, it's hard to defend pointing to a co-worker and calling their "heroic" behavior negligent. This is an industry that almost never follows its own recognized best practices, managers favor quick and dirty solutions over slightly more expensive solutions that have a lower TCO on time scales as short as months, and 5% of the work force carries the dead weight of the other 95%.
    – StackOverthrow
    Mar 26, 2020 at 17:17
3

That often improves based on experience.

Saw one case where the IT team asked for UPS on 3 sites for the servers based on data safety - did a report showing consequences etc Answer by management was NO...

2 months later one of the 3 sites got hit by lightning...data loss, time lost in rebuilding etc etc.

Within 2 weeks all 3 sites had UPS for the servers... a real case of shutting the stable door after the horse has bolted.

Or “that is such a tiny risk it will never happen” which is why many companies did not pay for pandemic risk insurance...

Improve this answer
1
  • 9
    That was a case of shutting three stable doors after one horse bolted, so it could have been worse.
    – gnasher729
    Mar 26, 2020 at 8:48
3

Research safety-critical industries like aviation. Planning ahead and mitigating threats is very much rewarded, and heroes can get in trouble (up to being fired or losing their certification) for not successfully anticipating problems or following procedures - even if they then "save the day". It's a learned culture that I've found very different from many other industries.

Major features of this culture are open lines of communication between levels in the hierarchy, a culture of compliance with procedures, but also giving experienced people on the team enough freedom to make decisions without interference from upper management. The most important tool is a briefing before and after each event (call it a team meeting if you like). The debrief, especially, focuses on 1) what was done well, 2) what was done poorly, 3) how to fix it going forward. For this to work, you need mature professionals who can point out problems, even their own, and who can give and accept criticism without taking it personally. You need leaders who can take criticism as well as they give it. A culture of both not assigning blame (including not giving out punitive punishment except for egregious/careless mistakes) and accepting responsibility for the outcome as a team is essential for this to work.

Improve this answer
1
  • Education plays a big part in such cultures. Most cultures (and sub-cultures) are too near the precipice to have time to be educated to stay away from the precipice... And communication needs community, rather than individualism.
    – Philip Oakley
    Mar 26, 2020 at 17:09
1

I am proposing a monetary solution. There is a project in Germany to pay the doctor for being healthy. People subscribe to a doctor and the insurance is paying him for every healthy person. If a person gets sick, the payment is lowered. The intention is to fix the health of the people in the long run, instead of monetary incentivising the doctor to fix the health for the short term, so that people have to return for another treatment.

To be honest, I don't have any experience with implementing this. My idea is to pay a bonus for IT maintenance that gets lowered for production errors or downtime. This should incentivse the maintenance crew to have highly available systems, learn from mistakes and implement prevention measurses.

Improve this answer
4
  • 1
    Incentives are almost always perverse. There can never be complete 'fairness' because the symptoms and the cause are not perfectly aligned. It's always possible to game such a system. Death has a way of making systems non-linear.
    – Philip Oakley
    Mar 26, 2020 at 17:05
  • 3
    There are utterly giant problems with these sorts of incentives. Why on earth would a doctor take on a patient with a bad medical history if they can be punished in this way.Or obese patients? Or smokers?Even if you legally mandated they can't do that they will always try to skirt the rules given such incentives.As @PhilipOakley says, these sorts of things always backfire, it's basically 'Goodhart's law'. Common analogues in programming would be to pay programmers to write bug-free code. Or to pay people to fix bugs. In both cases perverse incentives almost immediately cause big problems.
    – eps
    Mar 26, 2020 at 19:15
  • But then what do you do about people who try to choose patients who are already healthy?
    – Matthew Gaiser
    Mar 26, 2020 at 19:24
  • @MatthewGaiser maybe you call them Lawyers. Doctors don't train to treat the problems of the healthy.
    – Philip Oakley
    Mar 27, 2020 at 23:12
-1

I think you're confusing the work of individual workers and the job of management. Workers find or fix bugs, while managers create processes and plans to ensure that bugs are found and fixed. Admittedly this can get blurred when you're a senior developer dealing with non-technical executives, or a solo IT guy who is still seen as just an electronic janitor but is actually vital for the business.

The management decide on the balance between risk and reward. They decide whether to hold risk assessment meetings, which risks to mitigate and how.

It's OK for management to reward individual workers for heroics that save them from poor management planning, because it doesn't hide the fact that the manager failed to plan properly. Managers get rewarded when their department runs smoothly and hits their goals.

Improve this answer
2
  • The worker - management distinction doesn't help either if it ends up being used to reinforce (implicit) bad management (viscous circle vs virtuous circle).
    – Philip Oakley
    Mar 26, 2020 at 17:13
  • Managers are stuck in this problem like everyone else is. The manager who completely prevents a catastrophe is not rewarded like the manager whose team fixes the catastrophe. This problem is much deeper than simply having good planning, it goes to really basic things about human psychology.
    – eps
    Mar 26, 2020 at 19:28
-1

Give managers and workers a (financial) incentive based on uptime of the system. Perhaps a bonus, or promotion prospects. At a lower level, publicize the uptime of your systems internally.

At weekly or monthly meetings, mention the teams or systems without a downtime since the last meeting.

Or celebrate whenever a system reaches one year without unplanned downtimes, and with all planned downtimes ending smoothly. (It wouldn't do to incentivize deferred maintenance.)

Improve this answer
1
  • Goodhart's law: once you make uptime a target, it ceases to be a good metric. People are phenomenal at gaming metrics. Additionally, you have de-incentivized taking risks and making innovations, because that might lead to downtime.
    – eps
    Mar 26, 2020 at 19:21

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .