Staying organized as a research manager/team lead

Question

I work on a data science team in which most work is practical, data-oriented research usually terminating in the production of a prototype which is placed directly or with little modification into production as a data point for clients to buy. My area has grown substantially and I now have two and will soon have several more direct reports working on it with me. This is my first experience in management.

Previously, the number of tasks and avenues of exploration were few enough for me to keep track of mentally. This has turned into an entire research program, however, and I find it difficult to keep track of what hypotheses need to be tested, what new hypotheses are being generated, each person's results, etc.

I think this analogy captures my struggle: I'm responsible for dispatching forest firefighters on the ground, but it's a big forest and I'm on the ground too, so I can't coordinate everyone and see where they need to go next. I need a helicopter that lifts me above the trees so I can see the whole area, identify what paths each firefighter needs to take, and see new fires that have started. For me, this "helicopter" could be a visualization software, a whiteboard strategy, a personal habit, anything that helps me "map" out the entire forest so I can give orders that efficiently reach our goals and prevent me from losing sight of part of the forest. I used to be able to do this mentally, when the forest was just in a single valley, but it has grown to cover an entire mountain range.

Answer 1

It seems like the people on your team are fairly intelligent and hopefully want to do good work. Some see project tracking and other forms of documentation as drudgery, so work with them on finding a solution. It needs to be accessible (is a desktop app good enough or will they need access away from their work station) and is it easy enough to use. Ask for volunteers to find something they like and share it with the group. You'll make the ultimate decision but buy-in can be easier when others feel their concerns and input have been heard.

Many of the available systems have ways to report and summarize the data. You'll need to review several. Many apps have trial versions, so other than your time, there's no expense.

You could also look into Mind Mapping software.

Answer 2

Working on projects as a project manager, I've learned the reason certain tasks are hard to track is usually due to confusion with current work-flow process or people not understanding what their role in the overall plan may be, or over-complicating the project in general. Finally, there is the possibility that I have not clearly conveyed the required tasks and assignments.

There is no software or application that can fix people's broken processes including your own. Start with something simple. Could be a whiteboard in a common area that everyone sees. List out the tasks, and who is assigned to them. As they get completed, check them off. Meet every so often to see if folks need support or resources (find out what other work they have, or even life issues, that may be consuming their time). Eventually those tasks on the whiteboard not getting done and people assigned to them will stand out.

If people are responsive to something simple like that, and things are getting done, the next logical step would be to capture the same sort of thing in a similar application. Maybe even something as simple as a shared Excel file. There are many online applications too. I've also used things like smartsheet.com and basecamp.com but you do face the possibility of users balking at "yet another system" they have to log into and learn.

Answer 3

Organizing a research team is like organizing anything else. You need to know what your goals are and your progress towards those goals. There are only dozens of competing methodologies that you can adopt. Perhaps your employer would send you to a class in the corporate favorite? Management training is a pretty typical thing to receive along with management responsibility.

Answer 4

Data work in software is like any other software work. One thing critical to do is to make sure that all of the code you write is in source control.

This can help in several ways. First, if you need to change something already on production, you have a starting point. Second if an employee goes out on vacation or emergency leave or gets seriously ill (they shoudl be checking into a personal branch daily at a minimum), you have a record of what he or she was working on for someone else to use to continue the project. Third, the code can also often be used as starting point for new projects. Fourth, it makes it easy to code review everything and it ensures that only the official version is sent to production as none of your researchers should have direct rights on the production databases and code they write to query it should always be pushed by someone else for security reasons. Finally, if it is organized by type of project, you can look at what was checked in to see what lines of research have been worked on. Since alot of things won;t work out, I would suggest that all lines of research get checked in and that you havea separate location for deployments of things that are approved to go to production. Source control can also store documents as well as code. We put reuirements in there and some othere documents. IN your case, again , it is agood way to store onformation you collected for reseaech where others can see it and know what you did and did not do. Another critical item is that all code should be code reviewed. We review 100% of all code here including any database imports/exports/reporting queries/ analyses/changes. This ensures all are familar with more than their own code. IT helps people see different ways of approaching problems and learn new techniques and it catches a lot of errors before they go into production.

Jira can work fine as a ticketing system. What is critical though is that no one on the team including yourself, start to work on something that does not have a ticket. Ticketing systems are only as good as your willingnees to use them. YOu have to mdel good habits to your subordinates and you have to make sure they are not allowed to circumvent the system.

AS far as managing projects, generally I prefer to havea daily meeting where everyone says their progress and tell what they are planning to do. You might make it twice a week if you think daily is too intrusive. Meeting should not cover how to fix problems just state what roadblcoks they have run into and you can then get together with just the people involved in fixing the roadblock after the meeting. I recomment you spend no more than five minutes on each person's status in the status meeting and anythign that needs more depth be put off until after and then only includes the people directly affected.

Another thing to realize is that getting rid of their roadblocks is now more critical than any direct data analyses you want to do. You should never be assigned to the mroe ciritical taks becasue you will be frequently pulled off to make sure others can keep going.