It isn't fair between employees, nor is it going to be
Trying to make everything fair in a workplace is hard, if not impossible. Everything will look unfair in someone's eyes. So I think you need to lower your expectations in that regard.
Treating this as an issue of whether it's fair on you vs your coworkers will only sour your relationship with them, when it wasn't their decision or responsibility.
Address the issue - fairness in your relationship with the employer
The treatment of other employees is not the issue, and the family justification is really a matter between them and the employer. If they were disabled, had agoraphobia or there were some other thing inhibiting them then fairness between you and them wouldn't be something you could even consider.
The issue is that you don't want to do this trip, and don't feel you are getting reasonable treatment. Perhaps you hadn't expected so many trips, or they have increased in frequency since people had families. Whatever the reason, the issue is between you and your employer to resolve the problem. They need to perhaps pay you more, or offer you more time in lieu, or grant you some promotion in acknowledgement of your broader role.
Disgruntlement is not in the interests of the employer
If this situation gets worse, then relationships within the team will be compromised and you may be likely to leave for greener grass. None of that is what the employer wants. So it is in their interests to find a solution that works for you.
Only by raising the issue and coming to it prepared with your own position on what you might accept by way of compromise can you find an acceptable solution. Once the problem is apparent, they can decide whether they want to meet your position or adjust the balance of who goes on what trip.