Added later: I forgot the most important thing. Do not assume the role of the victim. This was a rude and annoying comment and that's all. Most probably he said this in an attempt to appear cool to you. Maybe even flirt. There exists this whole theory of flirt based on "negging" women that usually men do not understand how to use and end up just being rude.
I'm not saying this in order to excuse him. His motive doesn't matter, if you feel like he overstepped your boundary, enforce it. I'm just saying this to tell you not to be afraid.
And finally, don't be a bully. This may seem to contradict what I have written above, but it doesn't. Your replies should target only the behaviours that annoy you. Do not comment on his appearance, gender, wealth, irrelevant behaviour, etc. This should be a matter of principle, but it is also useful. Later if he realises that he was being rude to you, he will have nothing to hold it against you. So if he apologizes later, you should have nothing to apologize about.