It's interesting to look at questions like this, not so much because of the question on the face of it (which I think has been well answered by a number of the others), but by the underlying subtext. I'm going to quote some of the question back to you:
Coming from a very poor family I haven't traveled at all in my life and I don't own anything
I am a working professional making a decent living, am married, and have kids I suppose people assume I am financially comfortable
nowhere is the answer because I have a crippling student loan debt
they throw around conversations about 6k+ vacations which I have never had the privilege of partaking on
You and I know these people fall into 2 groups:
- People who are paying for this via credit, haven't learned from the last few years and are heading for bankruptcy
- People who CAN actually afford 6k+ holidays either by financial acumen or by having money
I won't mention the second group (enjoy your holiday!), dealing with what to tell the first group is covered elsewhere.
The thing to me is not the dealing with them, but the effect they have on you. Looking at the story you tell I can see things have been hard, and you have been doing the right thing by paying things off instead of mucking about having a good time.
BUT I think the reason for this question is deeper, the issue is more about looking at these people and thinking "what if they're right, and my family and I are wasting our lives doing nothing but paying debt?"
My answer to this is two fold, you are RIGHT to be focusing on the debt, but I think something else needs to be done.
Spending 10-20 years working hard, and doing without is missing quality time with your family, now I'm not saying "screw it, get the AMEX card out!", but I think there are 2 ways to go:
- The Dave Ramesey kinda thing: - Get "sick and tired of being sick and tired", get the whole family involved, take on as much work as you can (extra jobs, selling things etc), BUT set a realistic window (say 2 years) and throw EVERYTHING at the debt (I don't care if the student debt is at a low rate, you need to get rid of it asap). This becomes like a marathon run, as hard as it may feel for a while as you get closer to the goal you can keep pushing yourself to the finish as you know there is one. Once that's done you can relax and use the decent living to go on holidays etc that you can AFFORD, and will enjoy all the more due to the effort.
- Prioritise: - Realise that you can't keep on the death march for 10-20 years, start putting some money away (even if it slows down the overall finish), and take quality time with the family. An adventure can be 100 miles by bus and camping in a tent, do what you can pay for, but prioritise the family time, you'll have lots to talk about to the co-workers, and probably do better at work (so get the chance to earn more and pay things off better).
In 40 years time, you'll remember the family adventures, not the debt.
Either of these suggestions will give you something positive, and allow you to answer the co-workers without feeling bad.