You shouldn't feel bad for bringing up the issue, in fact, it's somewhat insulting that they either forget your dietary needs or worst, disregard them entierly.
You should approach the issue respectfully but firmly and ask for a change of vanue. Remind the organizer that you are a vegetarian and that the place they chose has no vegetarian options*. Also, you should also remind them that this is not the first time that the issue has come up (if it's the case) and that you would really appreciate it if the next time they will remember this and will choose a place that can accommodate your dietary needs from the start.
You should do it as fast as possible, don't wait until the last minute, when it may be hard to make any changes, give enough time in advance to make the change.
This lunch is paid for by the company and is part of your job, even if it's not mandatory participation, it's a bonding activity for the team and excluding people from it because of their dietary needs is the exact opposite of the desired result. A meal supplied by the company should accommodate all the workers and be inclusive.
* - unless it's the case, as some suggested that the organizer has made arrangements to have a vegetarian option beforehand. In any case, it's not your job to make sure that the lunch is inclusive to all, it's the responsibility of the organizer.