Let's say your boss is Betty and your new contact is Dave. A clever and efficient approach called the "BCC Switcheroo" is described here:
“Great to meet you, Dave!” you write. “Let’s definitely get together to talk about world domination. Betty, thanks for introducing us—I’m moving you to BCC so you don’t need to read all our scheduling emails.”
That is, you send your reply email with your boss on "blind carbon copy" (BCC). In this way, you can thank them and they know that you are following through. Yet this also automatically removes your boss from any subsequent messages, even if your new contact uses Reply All, because the email does not contain your boss's address as a recipient. All of this is accomplished with one email, rather than two as in Old_Lamplighter's answer.
Sometimes BCC is considered sneaky, but that does not apply here because you are explicitly noting in the email who is on BCC and why.
Your question addresses a scenario that is frequent and challenging enough that it was precisely the motivation of the article linked above, discussing this solution in detail.