What I take from your description:
- There is no way to reason him. You already tried and it's obvious this doesn't work out. So there is really no point in having the same tiring discussions with him again and again. He won't change.
- You also don't want to be silent, because beeing silent can be taken as tacit approval.
- As you are a contractor, there is no really no place to go, which has the authority to make him stop his behaviour, without having a high likelihood of backfiring and getting you into trouble instead.
Of course you could just search for another job, or tell your company to give you another position, but the truth is, that we will unfortunately meet a lot of those people in life and we don't always have the choice, to work with people who are likeable.
What I would therefor suggest in this situation - atleast if you are unwilling to change it by searching for another job - is showing brief disaffirmation, without engaging in a discussion, everytime this happens.
Some examples how you could behave:
- He makes a racist comment: You just point out "That's racist." nothing more.
- He makes a misogynistic joke: You just say (with a 100% serious face of course!) "That's not funny." and nothing more.
- He is berating woman: You just look at him with disrespect in your eyes and say nothing.
Of course this will be awkward at the beginning, but the goal here is to withstand the natural reflex in human behavior, to play over awkward situations by joining in the group or trying to change the topic. Let it be awkward. Awkward silence is actually a strong tool in making people realize, how silly they are. Just don't give him that confirmation, that it's okay what he is doing, because it's not!
Deciding on how to act:
While the above is in my opinion a sound mode of behavior, which enables you to face this situation with integrity, at the same time we also shouldn't ignore the fact, that we don't know how your manager will react to this.
In the best case you will be able to just keep on working without beeing bothered by this anymore, because he will just stop socially interacting with you in this way. In the worst case of a really toxic person, he may make your job really miserable to the extent of bullying and working to get you fired.
In the end, how to behave is your call to make and a lot of factors may play into this decision. (Would your company back you in case of a conflict? Do you have any job alternatives if the worst is going to happen?) Keeping your head down is certainly also an option, if you feel that you are not in a position to start a conflict at the moment.
Alternatively you may also want to try out the following, which is one of my favorite strategies in the case of racist/misogynistic jokes: Play dumb, say that you don't understand the joke and ask him to explain it. There actually is no way to explain a racist/misogynistic joke, without actually admitting beeing racist/misogynistic in the first place, which will make him look really silly. Of course you can't overdo this (it works better with peers instead of with your manager because of the power-balance), but I did that a few times in the past and had a good laugh at the people struggling to explain such a joke.