You handled this well. A colleague's understandable embarrassment at unintentionally committing a faux pas is not something you can control. If you have to decline a gift, whatever it is, you only have to do so professionally and respectfully and it sounds like you did that. It's not unsurprising that your colleague had an unpolished reaction to a rejected gift or he may have simply realised in the moment that alcohol in particular makes for a bad gift to people you don't know well. About the only thing you could or should do in that situation is to move on by changing the topic to something work-related or ending the conversation. After thanking your coworker for the gift of course.
It sounds like you didn't have any trouble with how to phrase your reply when you declined this gift, but the below is a general script you could follow, adapted from Alison Green:
This was very kind / thoughtful of you. You couldn't have known but I can't accept this [/ I don't drink] for personal reasons so I’m going to give this back to you and hope you’ll give it to a loved one or even use it yourself. You’ve already given me the best gift just by [your excellent input on Project X / being awesome to work with / being such an asset to the team]. Thank you and I hope [we can continue to work well together / I can count on you for Project Y / ...].
The key points are to express the appropriate amount of thankfulness, make it clear you can't and won't accept the gift, don't go into any detail about the reason that you're rejecting it, if possible thank them for a specific case where they helped you, and if relevant express that you'd like to continue your cooperation in the future.
The nature of the gift is ultimately not relevant. Plenty of cultures, religions or people have issues with accepting gifts. For other people in a similar situation: if you're willing to discuss the details on why you can't accept a gift you can do so but you never have to. "Personal reasons" is really enough. Anyone who keeps asking after that is simply being nosy to the point of rudeness.
However, my coworker appeared embarrassed
This seem more like he his embarrassed not because the declined gift, but because he brought it back bad memories. (Sure a lot of us have unfurtunly go through that)