It's really unlikely that not attending will actively damage your career. However, you could be missing opportunities to do better than you are now, and you could be leaving the wrong impression with some people.
I, too, have dreaded some of these social events. And sometimes when I go to them I really don't enjoy them. So I have sat down to learn two important things:
- what do I want from them?
- how can I increase my chances of getting that?
Buried in "what do I want" is "what do I want to avoid" by the way, and that's really important.
For me personally what I want from them is:
- to show I'm a "good sport" and a "team player" and someone who's willing to participate
- not to be stuck all alone feeling left out
- not to get trapped in a conversation about a topic I hate (politics, some kinds of sports, cars, etc)
- to meet some people I wouldn't otherwise meet, and learn a little about them
- to cause some people above me in the company to think slightly well of me (at least "a polite and pleasant person from what I can tell" if nothing else)
- to enjoy at least one of the "delights" - the food, the music, the prizes, the earnest speeches from management - that have been put on for us all
- to have the experience so that when someone refers back to something that happened at the party, I was there for it
How can I increase the chances of these things happening? For me, strategies include:
- form a team of 2 or 3 people and arrange to share a ride, or to walk together, or at least get them to agree they will be there. You can spend at least part of the evening with these people
- learn how to join small groups who are having a conversation
- learn how to leave a conversation (easier if it's not a two person conversation, but can still be done)
- try to be open hearted about the "delights" and see something good in them. Get out of conversations that are all about how dreary and lame and cheesy this whole thing is
- do not have more than one alcoholic drink eg one glass of wine
- check back against my list every 30 minutes or so and see if I'm ticking things off it. If not, get out of my current situation and go work on the list
Almost any book, website, tutorial etc on networking will teach you how to join small groups who appear to be having an interesting conversation, and how to politely end a conversation. This terrifies many people but honestly a simple "I'll have to ask you to excuse me, I'm afraid. Enjoy the rest of the party!" works very well and doesn't require you to pretend that your phone is ringing, you need to pee, or you need to refill your drink. Nor do you have to worry that your former conversation-mate will be mad when they see you in another conversation later.
Think of these parties as free networking practice. Assuming you don't throw up on the CEO, the stakes are very low. There may come a time in your career when an event like this holds tremendous value for your career. So learn how they work and what skills you need, and go this year and start working on those skills. You will get better every year and starting the first time you go, you'll see a benefit - people will stop asking why you're not going, for example.