Record Your Calls
First, I'm sorry that you're put in this position. This is a tough spot to be in. Chances are, your boss is thinking about the risks to the business and the costs of losing the vendor/client.
Right now, it's your word against the contact's, and if the contact continues to push this issue it can put you in a very tight spot where you can't negotiate properly with the contact because you're backed into a corner.
One possible solution, one that's more accessible than getting a new job, is to record your calls with the contact.
Fortunately, technology has provided us with quite a few tools that can be employed in situations like this to protect ourselves.
At my work, we built an app that gives you your own phone number, and with some configuration, it can actually sit behind your regular phone. Additionally, there are free tools, such as Google Voice that can let you forward your number to your Gmail chat.
The interesting thing about services such as these is that they record your calls. This is useful for several reasons, but in your case, they serve as a record of your conversations with the contact.
If this is not an option, talk to your IT department and see if they can put an "approved" solution in place. If you're using a VOIP phone service, this may be easy for them to setup.
One word of caution: I'm not sure what legal issues you'd run into. I think that as long as one party knows the conversation is being recorded, it's not illegal, and if it is, it may or may not be something you can use in court if the issue progresses. Your goal with using this tool should be just to protect yourself and show your boss and HR that you're not lying.
DISCLAIMER: I'm no lawyer, so be sure to check the last paragraph with a licensed attorney.