I work in California for a small company. I am a brown male. At one of the Annual year end party, I took my wife and my newborn girl to the party. I was introducing my newborn girl to everyone. One of the person (white male) around the table commented "Thank God, she doesn't look like you", and people around the table including my team member and owner of the company smirked or smiled looking at each other. I don't know if I should perceive this either a mean or a racist (on my color) remark. Perhaps he didn't mean it the way I am thinking. But it was pretty unsettling and upset me, I couldn't focus for the rest of the party.
The reason why this upsets me to so much lies in my past experiences. I have had to face the same remarks pretty much whole life (countless number of times) from a lot of different people including my friends and extended family. The same scene - people making remarks (direct/indirect) on my color, and people (even my closest friends) around smirking, smiling or laughing. It's like the people who are smiling or laughing are in agreement with this remark/joke, but they themselves wouldn't dare to make the same remark kind of thing. So when this person made the remark, I have had flashed of the past.
I am not a confrontational person, my anxiety levels go up when confronting a person. Am I overthinking about this remark? Should I just let this slide or report this to HR in email? If I report this to HR, I fear of the consequences such as having to deal with the mess, losing job and several other things.
Thank you.
Update - From the responses below, I gather that this is a cultural joke. I am from a different culture and continent, it's a highly offensive joke in our culture mostly directed at the skin color of the person, therefore we tend to be highly sensitive about it. Thanks for all the responses.