Is there a culture on the "business" side of the company demanding or expecting estimates to be small?
I know I've worked at a place where there was a lot of pressure for smaller estimated times, and it only caused a larger gap between estimated completion times and actual completion times.
What I do now (as an independent contractor) is take a feature, break it down into very small steps (about half hour to several hour chunks). Breaking tasks down into as many small tasks as possible will generate much more accurate estimates and will give you more tasks to explain why a feature will take long to your boss.
Make sure to include research and planning/re-planning time (people often forget those). The more uncertainty there is or the more research needs to be done, the larger the estimate must be padded. Also make sure to add items for unit testing, cross browser testing, documentation, deployment, demos, code review, a task for changes that may need to be done after completion (look at how long change requests usually take and apply it to this feature), management, meetings/communication, etc.
I use a personal wiki / outliner software called zim-wiki (similar to Evernote/Onenote/Notion/Obsidian/etc) and create an outline of tasks and subtasks multiple levels deep. I start the outline by copying/pasting a checklist template with common tasks like the above that I often forget to include. This is in addition to project management software like JIRA and is your personal plan for a JIRA task (you can add the outline or a summary of it to JIRA if you want).
Next, sort the tasks from highest risk/unknowns/difficulty to smallest. There have been so many times when I've went through a feature completing items in the "logical" order, sometimes doing easy/medium tasks first, and when I got to the hard parts, I realized that I had to completely redo the previous steps. This was because while working on the hard/unknown parts, I realized my approach was incorrect. Doing the hard/unknown parts first (to the extent this is possible for a given feature) is crucial. The easy parts will practically take care of themselves as you slide into finishing the feature.
Estimate every task at the lowest levels of your outline. Remember that you have a bias to be overly optimistic and estimate best case scenarios. Remember that your boss most likely treats your estimates as a deadline (i.e. close to worst case scenario). Use estimates that are about halfway between the average case and the worst case (adjust this as needed). Over enough tasks, this will even out to give you enough padding to complete features on time almost all the time. Round up each task to the half hour or hour. The only exception is that if I'm confident that a few tasks will take less than about 5 minutes, I might group them into one half hour chunk.
Then, sum up the hours at each higher level of the outline and after every summation, gut check the estimate - does this sound like a realistic estimate to do this ENTIRE task? If it doesn't, add some padding to the subtasks until it seems like you will have more than enough time to complete them. Add extra subtasks too if you come up with them. Tasks often take up more time than you expect and giving yourself a very comfortable amount of time to do them is important.
Common padding amounts range from doubling to quadrupling the time if you haven't done this task in this environment before. Consider padding even more if there's research or many unknowns involved. Whether to use specific multiplier(s) or not is a personal choice that you should experiment with for yourself and decide whether it works or not. I just use my gut feeling about whether I'm comfortable with the estimate. But the gut check must be from your gut, not your boss's gut.
If you think your boss will complain about how long it will take, look at your task breakdown and write down a few notes explaining why it will take this long - the cross browser testing for this feature will be tricky, the algorithm may need to be optimized, there are multiple possible solutions that must be researched and tested, etc. Have these ready when you tell your boss the estimate.
Then track your time as you work and adjust future estimates as needed repeating this process (add a half hour to an hour for re-planning/re-estimating when you do this). For time tracking, you can use a spreadsheet or software like Toggl, but also add actual times to your planning outline e.g. here's a sample task that took 3 hours even though it was estimated at 2 hours (I use a slash between them):
[*] (3/2) research library A for feature X
If it's taking longer than expected, communicate this to your boss as soon as you can. Do another estimate-gut-check and give your boss the updated realistic estimate. Don't give them a smaller estimate hoping you can catch up - in fact, consider adding additional padding so that you deliver on or before your new estimate. If your previous estimate was too low, it's almost certain that this one will be as well, so account for that with padding. Track this time re-estimating and communicating with your boss.
As you work, review your estimates and planning outline regularly. Repeat this process for every feature you work on until you get better.
I've noticed that using the estimate gut check and padding tasks until you feel very comfortable that you can complete them in that time has really helped me. And taking notes on why a task will take long and reviewing those before you speak to your boss has also helped.
At the end of the day, remember that accurate estimates will make your boss's life easier and if you can deliver on that, they should be happy.
I would stay with your current job and work on getting better at estimates. You're clearly valuable to the company and deserve the salary you're getting paid. Just keep improving your skills.
P.S. Here's an approach to padding estimates by multiplying them by a historical velocity ratio: https://www.joelonsoftware.com/2007/10/26/evidence-based-scheduling/
Other things you might consider doing:
- Ship smaller features for more accurate estimates (what is the smallest independent unit of a feature you can ship?).
- Shift to a shorter sprint cycle like a 1 or 2 week cycle (if you do this, make sure sprint overhead is low enough to accommodate this).
- At the beginning of a project and also if it's running late, cut scope, not time estimates whenever possible. It's almost always better to ship fewer higher priority features on time than to ship more lower priority features late. Scope can be much more flexible than it seems.