My recommendation would be honesty.

While you can't really control 100% how someone will take it, saying "hey, it's nice seeing you randomly in the gym, I enjoyed chatting the other day. But I really like keeping my own workout schedule and working out with a custom routine on my own." is totally fair and absolutely true. Keep it about you - you like your own routine, you like the chance to get away from work people and work thoughts when you're working out. You don't need to make any accusations about his behavior here, it's really just your preference, which is valid and justified.

I think you're right to not use short term avoidance, as working out is a regular enough pattern that you'd have to be clear about your preference sooner or later.

You can't really control his expectations and his reaction, but as long as you can make it really clear that it's not a personal thing, is a preference about how you work out, I think you'd be OK with the majority of bosses.

It's probably best to find a casual time - stop by his office when he's alone and his door is open, find few minutes after a meeting when it's just the two of you - something not too loaded and formal, but not public either. It was a casual invite so it can be a causal refusal.

My recommendation would be honesty.

While you can't really control 100% how someone will take it, saying "Hey, it's nice seeing you randomly in the gym, I enjoyed chatting the other day. But I really like keeping my own workout schedule and working out with a custom routine on my own." is totally fair and absolutely true. Keep it about you—you like your own routine, you like the chance to get away from work people and work thoughts when you're working out. You don't need to make any accusations about his behavior here, it's really just your preference, which is valid and justified.

I think you're right to not use short term avoidance, as working out is a regular enough pattern that you'd have to be clear about your preference sooner or later.

You can't really control his expectations and his reaction, but as long as you can make it really clear that it's not a personal thing, is a preference about how you work out, I think you'd be OK with the majority of bosses.

It's probably best to find a casual time—stop by his office when he's alone and his door is open, find few minutes after a meeting when it's just the two of you - something not too loaded and formal, but not public either. It was a casual invite so it can be a casual refusal.