It's hard to say. It depends on how you go about asking, what you are prepared to do for the team that's valuable to them while they are traveling and how much flexibility is built into your refusal to go. And also whether you and the team have a good relationship. And what the back story is behind your desire not to go.
"Nothing crucial for me neither, I'm NOT a key player of the team." Your opinion does not matter. What matters is the opinion of the team and of your management. They are the ones who have to agree that it is not crucial for the team and for the management that you be participating on that trip. If their opinion is at variance with yours, then their opinion wins. Even if you are not a key member of the team, they still might have their own reasons for wanting you to go.
If you are simply using the fact that your child is under 4 so that you don't have to travel, that won't go over well with your other team members and your management, a number of whom probably have children under 4. And note that these other team members must make their own day care arrangements for the trip. Having said that, if I were a single parent, I'd simply say "Unless it is important to you - team and management - that I be on the trip, I'd rather not go because I am a single parent and no one is really around if my child has an emergency".
Even if you are not a single parent, day care arrangements have been known to fall through. Children can also get sick and they can get hurt at the last minute, etc. As an Asian American, I don't recommend two-parent families as the ideal day care arrangement under all circumstances - we usually have an array of siblings, older relatives, in-laws and even nieces and nephews who are willing to step in and help. Ditto for Hispanic families. I think anyone who has children under 4 and goes on a trip - that someone is really hoping that things go well while they are absent from home.
If you are willing to back the entire team while they are gone (*), that will make a difference in how they perceive your desire not to go. Refusal to go is a different animal because refusal reflects an uncompromising attitude, and an uncompromising attitude is unlikely to be perceived as anything but unhelpful. If the team and management perceive that they make sacrifices and take risks while you are not, that may color their perception of your refusal to go and by extension, of you.
(*) As the stay-back member of the team, you could offer to be the contact for family emergencies for the team and offer as much assistance as possible to their families while the team is traveling.