Definitely(!) talk to your manager.
And in the meantime ... try to "spend a few minutes in this other guy's moccasins!" He's got a lot of experience – maybe he's genuinely just trying to share it with you. Maybe he's giving you very specific direction because he knows that without it you might not have a clue what to do. Maybe he's doing his best to try to help you.
However ... as you say, what's happening is: "You are terrified." "Your heart pumps harder." Okay, "that's a situation which calls for managerial intervention. This is a fundamental part of any "manager's" job description. (Trust me on this ...)
Without delay, ask to speak to your manager privately. Describe your situation to him or her exactly as you have done here. Don't apologize for how you feel right now ... just describe it. And then, let the manager deal with it: that's a fundamental part of the job. He or she will take your colleague aside and – also privately – discuss it. (So that your colleague is aware, quite possibly for the very first time.) And then, your manager might choose to have a three-way meeting to set a new path forward.
But: do not delay. Reveal, in private, that the situation exists.