This may be a difficult question to answer, simply because every company culture is unique. On top of that you have cultural norms for the region/state/country, and then even more detailed norms within individual departments or teams.
Back when I had the opportunity to work somewhere a few minutes from home in Southeast Texas, I often went home for my lunch break - far more often than you do (2-3 times a week). The company culture there encouraged a healthy balance between work & personal lives, however, and never had a problem with this behavior as long as I was back at my desk when expected. It also helped that many other employees lived nearby and did the same thing. This was at a company of roughly 1200 or so employees, with at least half of them located where I was.
Now... all that said, if your manager is making comments about it, then it's obviously an issue or concern for him/her. IF you want a healthy career from that job, you need to try to understand WHY it's an issue. If it's really just a matter of not being "a team player", then you should make an effort to attend lunch with the team occasionally. Maybe if the issue is your time away from the office, then a compromise would be to bring lunch from home as others have mentioned.
I do agree with other commenters that a lunch break is YOUR time, not the company's, but realistically the requirements of your job are determined by your supervisor and/or HR resource on an ongoing basis. Unless you're really determined to fight them on it, perhaps even in court, the practical needs of your role on the team are whatever your boss says they are.
If attending the team lunch is really that big of an issue for you, and not simply an inconvenience, then you should probably start looking for a new company culture. Otherwise, the best choice is to speak with your boss about the comments, but also just go to a team lunch once a month or however often makes your boss happier. You never know - you may even learn to like it.