Don't explain. Don't justify yourself. People decline jobs all the time.
If you don't feel like telling your friend the entire truth, then don't. You have the right not to. Besides, even a truth you hold can have many layers. You can tell him a little white lie or two, or you can choose to be vaguely truthful. Justifying yourself to everyone is just giving your power away.
Do you think that they'll tell you the truth if that potential employer decides not to hire you. Of course not. We're all human beings. We all have hidden biases. And we're all imperfect. Every employer would get sued if they'd let their employees tell the truth when they're rejecting someone.