There are some times when you need to ok things with your boss, such as when your "side gig" is competing or even similar to your day job. There might be clauses preventing a conflict of interest or potential IP issues with working too closely with your day job.
There are also job (like certain government jobs) that try to claim anything you do, even outside of work, as their own or at least want first dibs on it.
If it's just the manager's rules/feelings, then that's their own problem, but if it's the company rules, then it can be a real problem.
Talk to your boss/manager first to see if there's a company policy that prevents you from doing this. If not, you're fine and shouldn't worry about your boss. If there is a problem, you may need to get a lawyer involved. Involving that lawyer may just mean asking them if the company is actually allowed to have their clause and if it really affects you.
I had a previous employer that wanted first dibs on everything I created. The problem lay in that I had an existing side business and was continuing to create new products for it, which I'm still doing today after leaving that position. I had to do a lot of paperwork, deep history dives, and submit a lot of proof that my business was pre-existing to me becoming an employee and that anything I did for that business would be mine, as long as I didn't do it on their company hours.
I started that business while working elsewhere that didn't have any problems with it and even allowed me to get packages at my day job. It was a small company, so it wasn't really wasting anyone's time.
So, my point is: different companies have different rules. If you want to keep your day job, follow their rules or get a different day job. Or, as someone mentioned in the comments to your post, make your side gig your full time gig. That's what I'm trying.