1 - Speak to the manager privately
Whenever you have to give delicate feedback, it's wise to do it privately. Double that when you are giving it to a superior. "Delicate" can be anything you think might be taken negative. "Praise in public, punish in private" is a good guideline, even when the "punishment" is just saying "hey, please don't do that".
Since you do enjoy participating in the group bonding activity, be clear that you do want it to continue, you just don't want to eat allergens that make you ill. A good strategy is to clearly point out the problem foods and offer safe alternatives.
If you are very sensitive, it's also wise to point out what the limits are - for example, with a contact allergy, just having someone who's touched the danger-food put their fingers on your skin could trigger a response.
2 - Fix the food procurement process
Can be problematic depending on how your logistics work. If, for example, this is food brought in by the company, it may not be so easy to change. But if this is food brought in by an individual in the team, it may be that you can change the ingredients without a big hassle. If you have a reasonably straightforward allergy, it's OK to say - "hey, I can't eat --this food--, can we avoid that in shared meals?" - This is rather like the trend in the US for elementary schools to say "please don't bring peanuts -- too many kids are allergic" - they often ask parents not to pack nuts in individual lunches, because kids are so prone to sharing, that the school doesn't want to deal with the surprise when nut-bearing kid shares with nut-allergic kid.
Keep in mind that this starts to be less of a good plan if your restrictions are very complicated. For example, many gluten intolerant folks will just back away from food sharing activities, because it's pretty hard to figure out how to avoid gluten if you are new to gluten-free eating.
This varies as much by culture as by allergy. For example, you won't see much cheese in some cuisines. In others, it can be as if the very definition of "edible" includes "has cheese in it".
3 - Avoid it
It sounds like your reaction is not too severe, and you really enjoy participating. But if you can't offer simple guidance that people could reasonably follow, you may need to avoid the activity. There is a point where you have to make the judgement call that your issues mean a severe imposition to others. The golden rule ia a good guideline - if you would find your food-allergies to be extremely annoying, then you may be right.