I'm in a subject matter expert role at an IT company, currently I am tasked with a complex data migration which involves automated reformatting and quite a bit of partially manual, partially automated curation. I was chosen because I, as opposed to others in my role, also have experience in software development (not that much but I get done what needs to happen).

This has been going on for weeks and my scripts have been growing and growing and at every step of the way, more flaws in the source data become apparent, adding more and more error catching and detours in processing. With the added time pressure, the project code started incurring a growing technical debt - being absent due to sick leave didn't help with keeping a overview on things either, neither does working in a language none of my colleagues have experience in.

Since this type of work is a passion for me, this is somewhat personal for me and being, and being seen as, competent in these matters is important to me.

What should I do to get this back on track and ensure, most of all, that there are no mistakes introduced which are not immediately apparent?

  • 9
    1. Write tests 2. Get over it and talk to your manager about it
    – mxyzplk
    Feb 21 at 14:38
  • 3
    As long as you keep the original data source you will not lose any data, so don't worry.
    – Maxime
    Feb 21 at 15:20
  • 1
    @PM77-1: I presume typo for "this"
    – keshlam
    Feb 21 at 17:06
  • 2
    "my scripts have been growing and growing and at every step of the way, the source data throws a new unexpected thing at me, adding more and more error catching and detours in processing" - Welcome to software. Feb 21 at 17:39
  • 8
    “ the source data throws a new unexpected thing at me, adding more and more error catching and detours in processing” - Don’t think this way. If you only code for the happy path, and everything else is a (handled) exception, your code will quickly become an unmanageable mess. The source data is giving you new requirements. Code for them properly, add comments and tests, and refactor when the complexity gets too high. Feb 21 at 18:13

5 Answers 5


Advice given to novice pilots: if lost, climb, and confess. Get out of the way of others if you can, and ask for help.

It is better for management to know your status in time for them to help you, rather than waiting for the last moment and leaving them with no recovery path. If you are struggling, don't panic, but do let your manager know what you are struggling with. They may be able to advise you of a better solution, or tell you who to talk to to get useful input, or simply tell you that they understand that the deadline may have to slip.

Or it may be a serious problem, but even in that case they need to know, so they can coordinate that delay with others. You can get in much more trouble by unexpectedly missing a deadline then by saying that you are at risk of missing it. Keep your manager informed of both progress and challenges.

And don't be afraid to talk through the problem with coworkers. You aren't expected to be perfect. You are expected to keep trying until you find the solution, or until you establish that you really cannot find the solution and will need help.

Remember that if this was easy, they wouldn't need to hire people like you.

Meanwhile, remember to work in a test bed rather than on the live data, so any mistakes you make won't harm anyone else and are easy to recover from by just copying a new test bed. Try things. If they don't work, use the debugger to figure out why they aren't working and refine them or try something else. If you are out of ideas, talk to coworkers, including your manager; explain what you've tried and why it isn't working. In fact, simply the act of explaining it may help you understand and resolve the issue; this is the basis for what is known as "rubber ducking" a problem.

Stress is not all bad; you can't get excited without some stress. But don't let the stress take over.

Remember, too, that estimating completion time is an art and skill of its own. I have learned that for non-trivial efforts, I often need to double my instinctive estimate of how long the job will take, since that first estimate doesn't usually allow for things like weekends, other job obligations, interruptions, and so on. You learn to make these adjustments with practice.

  • Thank you, indeed, nobody expected perfection. I am absolutely not a novice pilot but somehow I never had a situation like this. Usually, when things are tough, I just push through and things work out or I adjust the plan. This thread helped to see the wood for the trees again. We split up the migration a bit and are in a much better place now. The people involved quite readily accepted that the work was way more extensive that anticipated... This was a valuable lesson. Mar 8 at 8:00

Trying to do a migration as one big event with no problems is an impossible job. This is true for data migrations, migrations to a new program, or even moving people to a new office.

Every data migration will have data loss and some trashing of the data. Be prepared to go back to the original data and redo part of the migration.

A better way is to do the migration in stages where you can manage the changes, the data loss, and the number of times to go back and recover missing data. The more stages and the smaller each stage, the more likely that the process will be managed well.

Talk to your manager about splitting this project up into multiple stages.

  • +1 for doing it in stages, don't try to do it all in one-shot! One stage may just be initial import of raw data and then progressive stages of cleaning and transformations. All restartable / chunkable.
    – DJ.
    Feb 21 at 23:40

Without knowing the exact coding and requirements, I am not sure how workable it would be, but you can still try:

  • You can ask for code improvements on Stackoverflow as long as the question shows effort on your part. Ask as many questions as you like.
  • Write as many test cases as you can, try to get all possible use cases. If possible, get end user perspective, that helps to come up with robust code.
  • Set up staging Environment and ask the customer's end users to test it out there and provide you feedback.

And the most important thing is that you should inform your manager on this. Try to explain it to him the struggles you're facing and in need of more time, if required. If he's a professional, he would understand that a half-baked product is likely to cause more problems.

And finally, stressing over work related problems is normal. We've all been here at some point of life, and this may also not be the last time you're in. You need to focus that only you, can help yourself in finding solutions for this by letting your situation known to your supervisor.

  • 2
    And stressing too much can lead to burn out.
    – PM 77-1
    Feb 21 at 16:19

I am assuming that you are not the only one who wants this project to succeed.

So identify stakeholders (hopefully your manager is one of them) and explain the difficulties you have. Get as much as possible into the open.

You may want to consider possible project scope narrowing or timeline extension.

The main thing get the real situation into the open. Arrange that users fully (or as much as possible) validate their data.

Basically, share the responsibility. Take some load off. There is nothing shameful about it.

Avoid getting yourself cornered into a burnout.

While it's not easy it surely beats the alternative.

  • 1
    +1. If you're digging into live production data (particularly relatively unstructured data) then there could be all sorts of problems lurking, which you are being responsible about by working carefully and systematically. Your stakeholders need to be aware that this is what you're doing, it is a much bigger job than they envisaged, and the consequences of rushing are very likely much worse for end users than any delays. If they tell you that deadlines trump data integrity, it sucks badly but on their heads be it... Feb 21 at 17:09

What should I do?

Sounds like you have already answered your own question:

> help to get me back on track.

Get help from a programmer who knows how to perform such a task correctly - migration of a large body of legacy data is a difficult, sometimes mission critical task. Your bosses need to understand that. Either inform them that you need help with what you have been tasked, or if that will compromise your position as subject matter expert, hire a free lancer (maybe you need more than a scripting language) on the side to help you. (If you take that step, be careful about publicizing it - you don't want to compromise your reputation as a subject matter expert.)

It is something I want to be good at this but am realizing I just may not be.

Not necessarily - you can learn and grow with experience and education. If you do get some help from someone more knowledgeable, make sure to tap their brain!

I can't have a code buddy because I used an uncommon language to do the whole thing (perhaps my biggest regret about this).

That would depend on the language, and how deep you are into the project. Some languages are fairly easy to port/translate, especially if you haven't gone too far - others, not so much.

Not the answer you're looking for? Browse other questions tagged .