If the company can afford for me to spend an hour during work time playing ping-pong, then I would much rather work for that hour and take off an hour early - that way I have more time for the things I actually enjoy doing.
I think everyone feels the same way about these things, including your co-workers. It may be that they are bored or trying to relax and they happen to do it by talking about golf scores, reality shows, or playing ping pong.
What I'm trying to say is we're all in the same boat. I'm sure at the time to leave, everyone punches out immediately, not caring about the earlier conversation. While you might feel like you're "forced" to endure their small talk, or that you feel like you're wasting company time, it may be that everyone is feeling the same way.
A good example is at this company's get together I had years ago. It was a "fun day" of pool games, drinks, and free food. Everyone was having a blast talking, and having a great time. They said folks could leave at 3pm. We had to do this tournament where the best two pool champs would compete. Guess what happened right at 3pm? Everyone just dropped what their doing, and left. Nobody stuck around to play or watch the pool championship. Why? Because nobody wanted to be there but as painful as it was, they did because they had to.