During your one on one with your manager (which is a red flag if you're not having), bring it up. There's a few different routes to take, which have different advantages depending on your manager and your personality:
- Focus on the team - by focusing on your team, you provide the image that you have their best interests at heart, which in turn will make your boss start discussing how these meetings help your team. A good tact here is to focus on "team members need a single voice directing them". Nobody likes having many bosses, and having different (occasionally conflicting) direction is bad for the company.
- Focus on problems - by asking your boss what problem he's trying to solve (communication is poor? can he not evaluate your ability? has someone complained?) you can work to solve the problem rather than poo-poo your boss' solution for it.
- Focus on communication - by focusing on you not knowing that this was going on, you're presenting a problem for your boss to solve (ideally with a proposed solution). If your boss loves solving problems, this can be a quick and easy way to get in the loop.
The main thing here is not to be angry about things, or to assign blame. And whatever you do, avoid being possessive of "your" team. Protective is okay, as long as the focus is on doing the best thing for the team (and by extension, the company). Possessive is not.
The goal is to point out that this sort of thing can be detrimental to the team (and by extension, the company), and that you'd like to compromise on something better. You don't want him confusing your team or undermining you or sucking up all their time. Your manager wants... something. Start by finding out what that is.