In the end only you can say how bad the situation is, and the degree of physical interaction verging on violence, and your perception are going to be entirely specific to you.
Here's some rules of thumb I'd suggest...
Seriously Unsafe
If you honestly believe that being in your workplace tomorrow will cause you to be injured physically, don't go to work. Go to human resources or legal and report an unsafe workplace. State that you refuse to return to the workplace until the safety issue is addressed. If they say "work or quit" - then quit and look into the procedures in your locale for filing a suit for failure to ensure employee safety.
From what you say, I'm not sure that this is the case for you - you mention some pretty physical rough housing, but nothing sounds like it's being done with the intention to hurt others, nor is anything so risky that you stand a serious chance of being hurt. But I can't tell from the outside, conclusively.
Cases where I'd consider this an option:
- Anyone in the office is truly physically threatened
- Anyone operating dangerous tools appears to be drunk, stoned or otherwise incapacitated
- Anyone in the office is encouraged by a supervisor to ignore basic safety precautions
Really, really demeaning
Say something.
If you have:
1 - Expressed clearly that you don't want to be included in the physical interaction (publicly or privately)
2 - Been ignored by those you spoke to
Then you have grounds to file a complaint with human resources or legal. Company policy and country norms will vary on this one, but at least in the US, most companies have an escalation procedure. Rules can vary whether the issue is coming from the supervisor, coworkers or both - but there is generally an escalation procedure that involves documenting the physical interaction, the statements you've made asking not to participate in the interaction, and then the fact that your request was ignored.
If your HR department isn't offering a procedure for reporting this, then take it up with your boss's boss and keep working your way up the chain.
I wish I could promise that using these reporting procedures would guarantee a positive result, but I can't. I think companies are getting smarter about the fact that ways of showing camaraderie are highly variable and what one person or group thinks is totally normal can be totally offensive to another person or group. It should be OK for someone to express the problem and for the team to find a way to include them in a less offensive way in a team spirit that everyone can pariticipate in... but that's a happy dream world that isn't realized in every situation.
NOTE: Fixing this can take days or weeks or even months. It's your call on what feels unsafe to you, and how much you need this job, and how much time you want to spend job hunting vs. working through this reporting process. If you don't report inappropriate physical interactions outside the sphere of the problem, it's unlikely that the problem will get fixed.
That's why the judgement call on safety - unsafe is one thing, uncomfortable, humiliating and inappropriate are not OK, but not the same thing as actual bodily injury.