You can just tell your coworkers what you said here.
Of course, most people don't like feeling like as people they don't matter, so there's significant risk associated with this that cannot really be removed, because no matter how you phrase it, people don't like being rejected and ignored socially.
In other words, I don't care about "company culture"
I think you are underestimating a few things.
First, understand that if you don't care about company culture and actively avoid participating in relationship building, you are accepting that you will very likely:
- Be seen as less of a team player than others
- Be less likely to receive promotions (particularly if work conversations do happen in some percentage of the time)
- Develop a negative reputation for being antisocial
- Especially if you leave early rather than socialize with your coworkers
- Have weaker relationships with coworkers
This is because, well, you've decided your coworkers don't matter to you and you don't participate in a fairly meaningful part of the company culture to many of your fellow employees.
Second, if this feels unfair... well, company culture is a rather important part of job "fit" - it sounds like you are at a company where you don't have a good fit. While this doesn't mean you should quit your job it is something to consider in the future. Make sure you focus on understanding this sort of thing in interviews. Luckily, most companies are very eager to be excited about a culture that involves ping-pong, etc, so you should have minimal problems identifying companies that are a lot more "work only" culture.
A lot of people have fairly meaningful relationships in the workplace. It sounds like you are not one of them. I would encourage you to take care when interviewing to understand the company culture so you don't end up in situations like this.
And third, you don't have to attend every social event to have a fairly meaningful mitigation of a lot of the risks involved and still receive benefits. I strongly recommend making an effort to participate at least once a week. This will mitigate much of the risk involved with straight up rejecting all social activity at work.
Politely declining socializing/games on Monday-Thursday feels a lot better to most people if they know you always are there on Friday, for example.
If you have to consider 30 minutes a week of socializing part of your job description to do this without being bitter, I would try to see it this way.
Last, it's important to recognize that relationship building is a meaningful work activity. It might not be to you (though I'm skeptical it has zero effect). But it definitely is to the overwhelming majority of people in the world.