I think this is a problem with your manager who's taking things a bit too personally. I don't think you need to continue apologizing or trying to make amends - you apologized in an email and that is enough.
What I would do is note for future reference that he does not take this kind of humor as you do, and avoid these sorts of jokes in future.
