Intrapersonal dynamics are a rough sea to navigate. To help here, lets break down what we know about the situation:
1) When you arrived it was strongly suggested you have lunch with a specific group of folks.
2) This group of folks is tightly knit, and have accepted you.
The question is, if you don't personally want to spend your free time with the group, what do you have to lose by dis-engaging them?
Do you work with them on a daily basis? The fact someone suggested you lunch with them, and that they are tightly-knit suggests you do, and it is a good idea to keep on friendly terms with them. Who do you have a problem with? Is it the one person, or the entire group? What is it you don't like about the group? Are they too critical, or do you not like it when one person gets the rest fired up?
If you decide you have no practical use for them professionally, and that your disgust outweighs any real benefit you could gain, then you'll need to break-up with them. Coming out and telling them your real issues can potentially have dire ramifications for you. Rather, I would use other professional events to OBE (overcome by events) your time with them at lunch. Working from your desk, meeting with the boss, talking to you spouse on the phone are all great excuses. Then after some time, let them know you can't contine to be a regular at their table.
Personally, I have made the mistake of confronting similar issues, and its never worked out well. Good luck!