You'd better realise very quickly and very certainly that you're in a dangerous situation.
Right after that, you should start getting particularly angry that a little rat face is manipulating you with the premeditated, determined goal to suck you dry until he can spit out the last bits of dregs - all for his own entertainment. He is actively enjoying every moment and morsel of your predicament - and feeds his deranged mind off of it as would a vampire hunger for blood. When he's done with you, he'll find his next victim.
Read up - as a matter of urgency - on the Internet about narcissists and why they pose a very real threat in general, and specifically for you. You'll equally find videos on YouTube discussing this sort of thing - explaining what it is, the methods, the goals and how its possible that you are naturally almost defenceless against it; one of the very reasons for these biological rubbish thugs to be as successful as they are.
Narcissism is a perilous mental derangement. How deep this mind-rapist has already infiltrated, penetrated and reprogrammed / altered your mind and thinking, is the chilling words which you have typed yourself: I don't want to deny his requests as I'm a very soft hearted and polite man.
One of the aspects which you should immediately explore by yourself, is why you give in to his demands and mechanisms. You might simply abhor confrontation, be fascinated with his eloquence, find him generally attractive or likeable, derive some sort of reward from him paying you attention, be intrigued with the power plays he concocts or whatever. For the sake of self preservation, find out exactly what this person triggers inside of you and identify your specific weakness, so that you can start rigging a strategy for your defence.
Find every possible reason for not liking this person - as long as the reasons are real and valid - and constantly remind yourself of them.
It's not quite clear exactly what your physical work situation is, but the best would be to avoid any form of contact if not in the presence of other work people, preferably someone higher up in your work hierarchy such as your manager.
You HAVE to break this person's hold on you. The next thing you know, you've not only lost your job, but your reputation as well - possibly for a very long time. If you're married or attached - specifically having other dependants such as children - you are right now placing their well-being in great peril. It might be awkward or even a disgrace to acknowledge the power that this predator is exerting over you when you begin to right now find help, but it will be much worse to be scraped off of the pavement later on.
You've already identified the dangers, acknowledged and explored some of your vulnerabilities, and started looking for sources of help, such as asking your question here. Remain on course, intensify your efforts and neutralise the web of tentacles that has already began to be woven around you.
Time, lastly, is not your friend. Neither is kindness and politeness - except extreme kindness towards yourself and the people attached to you.