Don't take offense unless you have to...
Were you really offended or are you just wondering? Are you sure it was MEANT to be offensive? I've been in the position where people joke and are inconsiderate, or trying to be funny (and failing). A single time wouldn't get me worried about it, unless it was an egregious insult -- but this doesn't sound like it affected you that badly -- you're almost wondering if you should be offended. Was it a dumb joke, not told at your expense, but instead just flat? Does this person make similar, weak-humour attempts with other people? Maybe this is just them trying to break the ice... and maybe feeling just as awkward when they were trying to come off an jovial and friendly. Heck, it could even be a therapy assignment for the guy.
Maybe he was trying to play off the fact that you weren't interested in his great idea of you changing so much in your life on a whim just to be at a different location.
He might not even be aware that you paid any attention to what he said.
If it really bothers you, I can recommend taking a gentle approach and ask. But try to talk to him without any fear or assumptions that it was an intentional insult, a dig, or whatever. You're going to ask him for more information, but you're not going to assume that he going to answer any one way or another.
Wait until you can approach him in a low-key, casual way and ask if you can talk to him for a second -- preferably ask when you can do it reasonably privately... there's nothing like telling someone, in the middle of a group of people, "I need to talk to you." That just sounds omminous and challenging. Sometimes, people will react badly just because they think that need to "save face". It can be as simple as a quiet "hey, you got a second?", and turn a little away from anyone near by, for a private conversation. Unless he's dense, he'll know that you want to discuss something privately.
The less-stressful you can make it appear, the less he'll be thinking about who's watching, "what will so-and-so think about this? How should I act? Am I under attack? Should I be all 'official'?" Done correctly, he'll be thinking, "oh, he has something he wants to share with me..." and it's almost a reward to him that you're taking him into your confidence. Maybe you want his advice, or maybe you're taking him up on that great idea he had, etc.
Just ask him about it, without being offended. "I was trying to figure out what you meant when you said ..." It will help if you can bring yourself to actually be curious.
You can also express concern, "I was just wondering if everything's OK. The other day, you said ... " -- Meaning, "so, are we good?" but implying "you said something that seems out of character for you, and I was wondering if there's something 'wrong between us'".
I think that if this can be done casually and quietly, he'll wind up being able to think more about how what he said sounded instead of thinking about "this guy's trying to make me look bad." That's not what you're trying to do. You're just looking for some clarification, in a calm and curious way.
Many people aren't aware of the effect that their words have on others... some (a small number, I hope) also don't care what effect their words have. Some people, if questioned, will feel awkward and the need to protect -- in an offensive way. I think that the majority of people would care if something that they said was misunderstood or misintrepretted, especially if that something could be misunderstood as being purposely offensive. The less awkward you can make someone feel, the more they'll be able to listen without the need to defend. You're trying to get them on your side. By asking for clarification, or even expressing concern over THEIR feelings, you've given them a "good" part to play, and things may get cleared up quickly, with benefits all round.
You can be more direct, if you feel the need, "You said ... and I just wanted to know if you meant ..." but that is more likely to put a person on the defensive, and people in positions of power don't always deal well with being on the defensive. Sometimes, being on the defensive will make people behave less rationally. Sometimes, just calling someone's attention to something they did, without forcing them into a position where they feel the need to defend themselves, is enough to bring about corrections in behaviour.
One of the keys to all of the above is your not being on the defensive, either.
You're not trying to make the person look bad, you're not trying to blame or shame, you're not the "wounded party" (implying they're the bad guy).
I've also been in the position where people try to make someone the butt of a joke -- a little dig or sometimes a "joking" insult, or even a direct insult. If that's the case, the above can still work, but you can be less "curious" and more matter of fact. You still don't want to try to make the person look bad. Even badly behaving people have face to save. Being plain and matter-of-fact can help deal with this, especially when you're not taunting or threatening them.
If you ever get to the point where you really are, seriously, offended, then it may be beyond that "I want to discuss something with you" and it may be the point where you want to start documenting things. Calmly and clearly. And you don't have to have anything to do with the other person at that point other than behave professionally. And at that point, calmly approaching HR with your issue is professional. Calmly so you don't look like you're going to freak out and sue everybody - which makes the COMPANY feel defensive. You want the company to be on your side.
Oh, well, look at the time (length). I've gone on too long. I hope it turns out well, no matter what your actions. Do keep us apprised of the situation.