How to estimate tasks scope of which keeps sliding as I keep working on them?

Question

Background: I am working on a legacy code base. It is huge. I did not write it. I have done a lot of work on it, but more work remains. There are still unknown parts in there for me.

Scenario that happened recently:

• Sales requested a change to a product line. Namely, update percentage figures based on new coefficients. Engineering gave me a cheat sheet with what percentages should be, and what the coefficients are.
• Assuming that my work will be to replace coefficients and update percentage, I thought I will do just that.
• I estimated 1-2 days to complete that, thinking that it's 2-4 hours work, but knowing that something usually slows it down.
• It took me 2 weeks, plus there is more work to be done, because knowing of my initial estimate I took a lot of shortcuts that are still there to be cleaned up
• There is probably at least a week of work to be done. Maybe 2, to get things where they should be. So much for "2-4 hours of work" as originally thought.

What happened?

Scope just kept increasing. I didn't know this, but I kept learning of this as I did more work. I didn't know it, Sales didn't know it. Engineering didn't know it. Code was written by someone else. Engineering didn't know of different algorithms that had to be replaced. Sales didn't know code. I mean in retrospect I could have realized that "both algorithms needed to be replaced", but I couldn't have, since "replacing coefficients" would work as a drop-in replacement, without having to touch anything else, I thought.

It was not possible to simply update coefficients as I originally thought, because new coefficients had entirely new algorithm attached to them, incompatible with old ones. There were two algorithms in the code using those coefficients. One simple that always assumed percentage at 100% and thus a few shortcuts could be taken and formulas simplified, and one that varied percentage - no shortcuts would work, you'd have to write the complex formulas. Engineering originally gave me the cheat sheet for 100%, and we both did not consider the hard complete version. That slowed me down a lot.

So main things that slowed me down:

• not realizing that it's not a drop-in replacement of coefficients. I need to update the algorithm
• Wow, this algorithm is HUGE and I keep learning that there is more and more to it (cheat sheet, then excel sheet, then multiple excel sheets, new formulas, etc)
• once I did, I started updating one algorithm, without realizing there is another one
• once I realized there was another algorithm, made shortcuts for it to work as well
• after updating percentage and coefficients everything else kind of needed to be updated as well. Other formulas, other variables (essentially what I refer to as "the algorithm")

Basically, scope creep and I don't see reasonable ways to uncover all this without a round-table discussion of what the change actually is, and insight into codebase and the project. I guess if someone had enough insight to reasonable foresee all this, they should really be paid big money for their talent. With hindsight 20/20 I can make up an imaginary scenario where there is an estimation meeting and people go

• "what are the changes",
• "what are the complete changes, in the worst case that have to be done" (i.e. rewritten)
• "are there more places where this change needs to be done",
• "what is the hardest thing that needs to be done"

etc. Maybe in this case at least in part, this scope creep could be mitigated.

This happens a lot to me with this particular code base and this particular project. How can I mitigate this or how can I estimate better? Or how can I communicate better that scope keeps increasing as I keep learning of new things. How do I communicate with bosses when they try to hold me to original estimates ?

Answer 1

The simplest answer is: state your assumptions clearly in your estimate (no off the cuff estimates, period - those always are terrible!). As soon as you realize an assumption is violated, immediately inform your client. They may revise the work, change their mind, cancel the project, things may go on hold - and any of these is almost always better than the alternative!

Example Time!

Request: replace hallway light bulb.

Assumptions/work to be done: get ladder and replacement bulb from supply closet (10 minutes), remove light cover (5m), remove old bulb(2m), insert new bulb(2m), verify functionality (5m), replace cover (5m), return ladder (5m), dispose of bulb (5m). Add time for incidentals and sum, you estimate 1 hour.

Great, so then you get to work and you're out of replacement bulbs. You are told it'll take 2 days to get a new bulb, so you inform your client right then that your old estimate is now null and void. So you you revise your estimate with "added assumption that the bulb is received in 2 days - 2 days + 1 hour to replace bulb, plus + 15m of work done to discover problem and order new bulb".

You get the bulb, it's the right bulb! You continue your work, insert the new bulb, and...it doesn't work. Something else is wrong - the fixture, the switch, you don't know. Let your client know - on big jobs you may need to estimate how long it'll take for you to begin troubleshooting. You might estimate "oh, it'll be about an hour for me to trace down what went wrong assuming it's the fixture, switch, or bulb." Work is on hold pending this new work item completion - if you need to, revise estimate with the new hour of troubleshooting.

You test and as far as you can tell the switch works, the bulb works, the fixture seems fine, but electricity isn't getting to the fixture at all. Something is wrong "somewhere" - it could be a breaker, wiring, cosmic rays...you don't know. Inform the client, create a new estimate for the new task "discover cause of lighting failure", as if it was a new task with new sets of assumption and estimates. The original "change light bulb" task goes to being on hold, pending resolution of this new task. And on it goes...

---

If you do it more like this, you're estimates are rarely really, really wrong. You might discover more work needs to be done than you expected, or more things are broke than you realize, or there are discovered dependencies, etc, but you didn't know that! Your estimate was based upon what you knew at the time. If the world was the way you thought it was, everything would be fine - but it isn't, so you need to communicate that.

The huge advantage here is that you make it clear that it isn't anyone's fault - especially not yours! And your client can surprise you with things like, "oh jee, there's really no need to go to so much trouble, it's not worth it!" Maybe your client was dealing with that bulb being out for years and just figured it'd be nice to have it fixed - and you find this out BEFORE you take down the ceiling, rip open the walls, and rewire half the building.

One of many other advantages is if you do this you can get better at the parts you have control of, and you begin to get better at the process of estimating and communication. You can start to develop a sixth sense for when things can be easier or harder than expected, and for how much time you really need to spend to figure out what a job might entail.

On bigger jobs, you'll generally spend an estimated amount of time on, surprise, "estimation" (some call it discovery, research, surveying, or more fancy terms that they get paid very well to dream up). Sometimes you'll estimate how much time you'll need to estimate to come up with an estimate - that's when you know you've moved up in the world!

If all else fails, in the future when someone asks you to change a light bulb you'll be able to know exactly how to handle it - quit. ;)

Answer 2

Note: the larger the job,the more likely (and the more times) that something unexpected will happen. Part of developing this skill is learning to allow for that and build an appropriate scaling factor into your estimate. This comes with practice, if you take time to compared results with estimate and correct yourself.

Another thing to learn is how overoptimistic you tend to be. I've found through long experience that I should always double my first "gut instinct" estimate. Again, pay attention to how good your guesses were and correct for this before quoting.

There's also some time lost every time you step away from the project, since you lose some mental context/focus. Sometimes a break is useful or tasks have built-in delays which can be used to squeeze in other work, but one hour every day rarely equals the same number of hours as a concentrated block. That's actually part of the reason large tasks take longer than expected. Know your probable schedule and figure that into your numbers.

Remember the Mythical Man-Month effect. Adding people doesn't always produce results faster. Nine women can't produce a baby in one month. And even on jobs that can be divided up, there is training and communications overhead, so it doesn't scale linearly. Beyond a certain point, adding people may slow the project. Again, the only way I know to learn the tradeoffs is experience, though if you're aware of these issues you may be able to make a guess.

Finally, especially on larger projects, don't quote a time/date of delivery. Quote hours actually spent on this task. If you are interrupted by other demands, this will be delayed. Hours makes that explicit, helps you push back ("sure, I can do that; it will delay these, though"), gives a basis for corrected estimates, and generally makes the tradeoffs clearer.