I can relate to what you're saying. My husband and I are both non-Christians, but not atheists either.
We both work for big company names in the UK and find that adapting to our work culture is a give and take relationship. I never minded working over Christmas every year as we don't celebrate (hospitality so no closing hours), which gave celebrating employees time at home; and work would always be kind enough to give me time off for our own celebrations, even if this meant shuffling things around at busy periods for my sake.
As far as celebratory references or parties go, neither of us go out of our way to say 'Merry Christmas', or give gifts. But if someone says Merry Christmas, we will always reply in like. Same with gifts/cards. Just so not to be rude. We will participate in an office 'secret Santa' if they're doing one. And we'll attend the Christmas party for a short while before taking our leave - though we don't go every year and it's never been a problem.
Since Crimbo has acquired pagan, Christian and secular aspects through the ages, it is safe to say that people's beliefs don't adapt to Christmas, Christmas adapts to people's beliefs. Many people today would agree that it's about family, good food, letting loose etc.
However if adapting isn't something you feel comfortable doing, then I think it's best that you've opened up about it in the first instance (so long as you weren't rude or snobbish about it).
Even as adults, we can sometimes become cliquey and stand-offish when someone says or does something we don't expect or like. They might judge you for a while, but there will be a new office-gossip topic soon enough. Let it go. Your work ethic will speak for itself. Sometimes it's the case that we feel certain people are behaving in a certain way as a reflection of our own insecurities about something, and it may not actually be the case. But even if it is the case (that colleagues are acting weird), I'd just get on with things, be yourself and let things blow over. You are who you are and going to a Christmas party or not does not determine your value to the company, nor your ability to socialise and build ties. People will realise this in time.