Don't explain. Don't justify yourself. People decline job opportunities all the time. It's not a big deal.

Do you think that a potential employer will tell you the truth if they decide not to hire you. Of course not. We're all human beings. We all have hidden biases. And we're all imperfect. Every employer would get sued if they'd let their employees tell the truth when they're rejecting someone.
If you don't feel like telling your client the entire truth, then don't. 

You have the right not to. Besides, even a truth you hold can have many layers. Justifying yourself completely to everyone is just giving your power away. Trying to control exactly how that client will end up perceiving you is a futile endeavor.  

And if for some reason, you do try to justify yourself, try to stay away from using universal quantifiers like everyone, all of them, always, or never. These kinds of universal absolute quantifiers can easily trigger further arguments. And try not to label people, if you do really want to justify yourself, you can talk about your own limited experience and your own feelings, it's difficult for people to argue against feelings, and also, sharing your feelings about a group of people is usually more truthful than trying to paint an entire group of people with the same absolute label. 

When it comes down to it, we're all individuals and nobody likes to be labeled for the wrongs committed by people of the same gender, race, or nationality.