His actions are completely unacceptable. If this was someone on my team, as a manager, I would want to know about it so I could talk to them. Maybe they did not realize how they were making you feel, so a third party stepping in to explain could help out, especially when reminded of disciplinary actions if they continue.
While I disagree about "politely thank them" for inviting you to lunch, I also realize that being too confrontational could escalate the situation and make you feel uncomfortable. I would say that someone like that person needs to get a clear NO. But it might have to be in front of another person to alert attention for safety reasons.
At this point I would inform HR about the situation so they can keep an eye on him as he might do this to other employees.
Sorry about what you went through, you have a right to be respected at work and no one should make you feel unsafe.