It's vital to be a team player, but being a team player doesn't mean you have to roll over for everyone. Being a part of team lunches once in a while shows that you have the professional courtesy to be a part of the group, but that you recognize that you have your own priorities as well.
If you always take part, you run the risk of becoming entangled in groupthink. If you never take part, you run the risk of being alienated from the group entirely. Part of being a professional is knowing both how to lead and how to follow. Even the best managers have to listen to their subordinates, and team lunches are a great way to hear and be heard. Make the most of your time with them, and respectfully bow out when you can't do it. Staying removed but accessible gives you the ability to contradict ideas discussed at lunch if you feel they're a poor decision, as well as keeps you connected with your coworkers' lines of thinking.
Most of all, it's important to be honest: tell your team, "Sorry guys, but I'm on a budget." They don't have to know that the chili gave you intestinal distress or that you thought you saw the fish filets moving; being conscious of your own responsibilities is excuse enough.