In general you can't refuse work trips. Whether it's to get training on something, to visit a client, to attend the company's conference, or to come to head office to meet your colleagues, once management decides you should do it, you need to do it. Ideally they would make the travel experience pleasant - a flight on a reasonable airline, a reasonable hotel close to the office, a reasonable meal allowance - but not every company does this.
These kinds of trips are disruptive. They mess with your childcare and your evening personal life (whether that's your guitar lessons, your kid's soccer practice or your weekly poker night.) If you don't have another adult who can take over things related to your children, your ailing parents, or your pets, they may be almost impossible for you. Yet, you may have noticed, companies do them all the time. And not just "make the remote worker come to us" either - as I listed above there are tons of mandatory trips that people would rather not make. But they make them.
Why? Because they are a condition of employment. Same reason people wear suits or enter their workplace through metal detectors or make sure they qualify for security clearances. It sounds to me like your trip is a condition of employment. I find the phrase "guilt me into going" quite odd, by the way. This isn't your cousin's baby shower. It's a work meeting. The people who are listing the positives aren't lying: there are some positives for you in these meetings, and some for the company as well. That you personally don't feel those outweigh the negatives doesn't mean you don't have to come.
So, my advice to you is this. Ask yourself, if this meeting was a condition of employment, that I must attend to keep my job, do I want to keep my job or not? Really spend some time on that. Update your resume. Look at job listings, and see if you can get 100% remote, similar work, for similar money. Knowing your chances of getting other work, are you willing to quit over this? Know your answer.
Then contact your boss and say something like this:
I understand there are positives for me and the company to having me come meet everyone every few months. As I've told you before, there are a lot of negatives for me. I'm not going to list them again. [One of: they can't be solved by spending more money, it's about the disruption to my personal life / they could be solved if you would raise the budget to enable x, y, and z.] I am asking you now to tell me very clearly: is attending a meeting like this 4 times a year a condition of employment? Is my job on the line?
Then wait. Your boss may say no, we love having you work with us, we can work it out, I wish it was better for you, can you stand once a year? And presto, you've found the polite way to refuse the trip. More likely, your boss will (perhaps after stammering and stuttering and looking at the floor and saying golly gee shucks and trying not to answer) say yes. Take a big breath. Then say one of
Thankyou. I appreciate your clarity on this. I will come. I value working here and if I simply must incur these negatives to keep my job, I will.
or
Thankyou. I appreciate your clarity on this. I cannot come. I guess I will wait to hear if I still work here next week. I'm sorry.
And again, wait. Your boss may still fire you even for saying you'll only come to keep your job. (I hope not, but it's a slight chance.) And your boss may say if you feel that way, I guess you don't have to come, if you'd be willing to quit over it. (But may start looking for a way to replace you with someone who does as they're told.)
Long term, you can't really keep a job where you're doing some of it resentfully and they know that. But you may be able to keep it long enough to outlast the manager who wants these meetings (I am guessing you didn't have to do these in the past) or to get to a point in your life where you don't mind them so much.