What are effective ways for a remote team to learn from a team handling an emergency situation?

Question

When a team is dealing with a high-priority, time-sensitive issue such as a system outage or a production issue, how can a remote team learn from the team tackling the problem?

Imagine your development team is spread out over multiple cities: Seattle, WA and Portland, OR. The Portland team has more experience dealing with hot production issues and the Seattle team members are new to the team, and therefore are not as experienced dealing with hot production issues. In this scenario, what are effective ways for the Seattle team to learn from the Portland team while the hot production issue is being resolved?

Some ideas I've had are:

1. Communicate via Slack: the downside of this approach is that the devs who are fixing the issue may not have time to spare communicating what the issue is as they are investigating it.
2. Have a dedicated Google Hangout so that the Seattle team can watch the Portland team resolve the issue: this shows promise as it is the closest thing a distributed team has to "looking over someone's shoulder" and asking questions while the triage is happening.

Are there more effective approaches than this?

Answer 1

This is a great question.

First, recognize that there are potentially conflicting priorities here. You want:

• To resolve the issue as fast as possible
• To train the other team (who will be slower to resolve the issue)

This means you need to identify first which is most important. Is it a critical issue and any wasted time is super important? Your business needs will drive this. I will assume this is basically a critical issue for the rest of this answer where solving the issue immediately is the highest concern.

Before we talk about communication, though, you need to have a cohesive and organized plan for responding to issues. By that I mean identify a point person or location for each incident. That might be a good thing for your less experienced Seattle team to own - an incident commander from there to effectively organize the response, communicate it, etc.

Once you have a point person they are responsible for organizing the response. If your Seattle team is very inexperienced, you might want to have a backup/shadow type incident commander to work with them.

Communication can only engage both teams if you have a defined plan for response. An ad hoc mass chat ping "stuff broke all hands on deck!" isn't as effective.

What we've found to work is a combination of things, primarily a easily accessible chatroom and a video call. You are correct in identifying that people actively resolving the issue are unlikely to type out massive walls of text. But they can talk in a video call and someone else can type updates (this is an advantage of the incident commander(s) - they aren't actively engaged in fixing the issue).

The primary people to talk in the video call should be those actively resolving the issue. Not bystanders wondering "hey what's the status?" -- that chatter should 100% be directed to the chatroom.

Incidentally video is important too, as faces really help form relationships and trust. What you are basically trying to do is build both those between your teams so try to have video if at all possible.

I would also note that the experienced team will probably feel "it's just easier for us to do this" -- you may have to convince them the benefits of having more people capable of handling support issues or otherwise "bribe" them into believing. This will depend on your company culture, basically.

Have a dedicated Google Hangout so that the Seattle team can watch the Portland team resolve the issue

You want people on both teams to be working together to resolve the issue. How you identify these people is up to you, but you want experienced people as well as inexperienced people on the support/response team actively addressing the issue.

During the issue, try to let the inexperienced team be guided by the experienced team. Even if the experienced people are 100% doing all the problem solving, have the Seattle team actually be doing the responses. People will learn far more from doing than watching. Have the mentor team guide and shepherd the other team, understanding that they can take over if needed (for whatever reason).

Have some of the more junior people be responsible for liasoning with the incident commander (or whoever is "running" the response) to give updates. This has several benefits:

• They have to understand what is going on
• They don't make the busy people solving the issue stop working

By designating "point" and "support" people for your roles (see above in having a process) you provide natural situations where people can learn.

And most importantly, once the issue is resolved, do some sort of after action review to document:

1. What happened
2. Why did that happen
3. What are all the steps the team(s) took to investigate the issue
4. What ultimately solved the issue

If you don't do this, any wisdom and insight from the issue will not be persisted anywhere except the brains of those who were actively addressing the problem. And the next time you will have to learn everything over again.

Doing some sort of review/retro is probably one of the most important parts of this. It's a low stress chance for both the point/backups to discuss and identify how/why/what the experienced team did.

By documenting the issue, your less experienced team can also consume the documentation. You might also be able to reproduce the issue in a controlled environment too - this would be good practice.

Answer 2

Let's start by saying that during an emergency is almost never the best time to be doing knowledge transfer. In an emergency the focus is necessarily on fixing the problem in the shortest possible time. Any kind of shadowing, mentoring or working together is going to slow this process down. Also people are under pressure, and are not likely to be at their best with their people skills.

Let me present some possible strategies:

• Have your less experienced teams do drills. Present the less experienced teams with real hot production issues, but after they are resolved. Don't let them see the actual fix, but let them investigate the issue themselves as if it were an emergency. Make the experienced team available for help but not to give them the fix.
• There is no substitute for face to face. Have the remote teams (or some members of them) visit the experienced team for a while so they can see everything that is done. Once you have a few people at the other sites who can do this, they can teach the other members of the teams.
• Have the experienced teams talk through the process with the inexperienced teams, again not during a real emergency, using voice links and shared screens.
• If you really want to do this during the emergency, voice links, web cams, and screen sharing would be my tools of choice.

Answer 3

I'd suggest some in-person cross training. Slack, Hangouts, other messaging/online collaboration tools are great once a personal rapport is generated, but it can be difficult to get complex ideas across quickly, especially in times of stress.

Depending on the severity and complexity of the problem, an experienced team lead can walk the newer team through a production hotfix. Rotating the newer team members (in singles or small groups to avoid cliques) through the Portland office will also help meld the two teams.

@Enderland's discussion about Documentation is spot on, and important.