Background - what in the world is your boss thinking?
I had previously written this as a comment, but I think it's worth expanding into an answer:
Part of me wonders if the reason the boss is so insistent on the OP going is because the boss feels the OP is not part of the team at all given the "I don't want to do things with the team, ever" attitude. Some bosses assume "team building" fixes these issues..
Some things to keep in mind:
- Your boss probably thinks you hate the team/dislike everyone. People who go through efforts to not associate with a team generally have a poor reflection professionally (and personally). No one in a company wants to work with a lone ranger who never associates with other people. The only people who "get away" with this for a long time are people who are ridiculously good. Otherwise it starts to reflect poorly on them professionally.
- Team dynamics are important. How your entire team functions is important to other people. You may not believe it affects you (we'll get to that later) but a good team has a cohesive team feeling. Each team gets there differently because everyone is different.
- Your boss probably sees this as a problem to fix. It's likely that your attitude comes across as "I am too good for this team to go to this activity" and your boss probably wants to help you fix that.
I suspect part of the reason your boss is so insistent is that they feel you are antisocial and do not fit into the team fine (and perhaps even other members of the team feel this way, too).
Some bosses just assume that teambuilding/networking things are a magic fix to this problem. For some people, they are, for others.. well they aren't.
Background - what are people feeling?
Many people, especially technical people, like to think feelings don't exist.
I have repeatedly made it clear to my boss that I am not comfortable joining 'social' events in work over the past 9 months by not going to friday lunchtime and after work drinks and not going to the last social event that my boss booked in my absence without asking me.
Your boss probably feels this attitude makes it difficult for you to be a valuable member of the team. Depending on how extroverted or how much of a feeler your boss is, they may even feel this as a personal rejection. This is a real feeling they probably have.
Just the same as you feel anxiety about social events, your boss probably feels a similar level of discomfort from someone being very cold/distant from them (and the team).
What can you do?
Ok, so all the above is really important context. When you systematically go out of your way to reject every activity resembling a social activity for your team, you need to take care in doing so.
If you talk about this subject with your boss, it is probably going to be an awkward and difficult conversation. What will probably happen is you say something like, "I feel like I'm fine on the team, I don't know why I need teambuilding activities" and your boss says, "everyone things you are really distant/cold, I'm trying to help you with that."
My guess is you don't want to talk through this with your boss from this perspective again (sharing your anxiety about such issues, wanting to avoid events because of that).
So what I would recommend doing is one of the following:
- Make an effort to do some sort of social activities. Maybe every other lunch. This is part of being professional, you have to be able to work on a team and do things like this. It's a lot easier to turn down other activities like this if you do something with the team.
- Ask for more advance notification. Some people just don't like planning in advance. So say, "I cannot make this, as it's a bit last minute and I already have preapproved vacation." and that's that. I've done that before when someone insisted on scheduling me for work when I was unable to, after a while I just told her, "I am not going to be here for this because I am unable to work as I previously told you."
- Approach the HR/legal issues. This is probably a last resort, but you may have luck finding your company policy or relevant laws on this. This will definitely not put your relationship there on a good place, however.
Try to understand why your boss really is scheduling you for these activities, too. It's probably different than what your boss says as they may not even know. I suspect it's highly related to the above (unless your boss is just a jerk). You need to address those concerns when you talk about avoiding all these activities with your boss.
You also need to address how and why your boss feels it's ok to schedule a work activity that you need to go to during your approved time off work. In most companies this is a really bad thing for a manager to do. I would try to get an understanding of why your boss thinks this is ok.
And if your boss is just being a jerk, well then that's a good time to consider looking for a different job.
At this point I am not concerned with how people think of me as a person, only how they think of me as a professional
Just a note here: in the working world, these are very interrelated. It's difficult if not impossible to separate your professional image from your personal image.