It probably helps to state the recruiter's motivation: getting paid for placing candidates. In this case, the recruiter will get paid if
They can place someone else into the other job you've applied for. If the recruiter already has a relationship with the firm, you're giving your position away. If they don't already have a relationship, then if you let slip the name of the hiring manager, for example, then the recruiter will be able to get in touch with them. If you let slip details of interview questions, be sure these will get passed on to any of the recruiters' candidates who go for the same position. So on this count you're right not to want to discuss too many details with the recruiter.
They can place you into one of their jobs. In this sense, it is helpful, in a general way, for the recruiter to learn how you're doing elsewhere: what kinds of jobs you're applying for, what kinds of interviews you're doing well or badly in, and so on. Moreover, a good recruiter will be interested in timing, and will try to arrange that you get interviews and offers for their jobs at or before other jobs you're applying to. In this sense, transparency can work in your favor.
More generally, the economic function of a recruiter is to create a market, and to create the market they need to understand it and get information from sellers (people like you and me), as well as buyers (employers).
So to answer your specific questions.
1 Is there a diplomatic way for me to stop future interrogation regarding this matter and to regain transparency regarding what I am doing that is outside of our relationship?
As I said above, I don't think the fact that you're applying for other jobs is outside the relationship. But it isn't in your interests to let slip any of the details. My approach is to be vague, or simply ignore any requests for details. So, I might say, "I'm interviewing with company A next Friday, it's a second interview and they will let me know within a week after that.", even if they probed for more details. There's only so many times they're going to ask the same question if you reply in a vague way. (Think how politicians answer interview questions, by being continuously vague, evasive, or repeating the same thing.)
2 On a going forward basis, how do I best answer such a question initially while not sounding sketchy, unfriendly, or suspicious and limit my interaction with a recruiter only on the business with him/her without them asking me what else I am doing?
Once again, for the recruiter to do the best job of getting you into one of their jobs, they do need to know something about what you're doing, e.g. so they know how to schedule candidates' interviews. I say something along the lines of "I'm applying to a firm who produce widgets, I got through the first interview, but contacted them to say I didn't want to go further at this point." I'd be happy to mention the name of the firm, unless the firm only accepted direct applications, in which case I would be opening up that firm to cold calls (which would be discourteous of me, and might make me look bad if that firm discovered it was me that let the information slip).
I think you do need to be sketchy, but this isn't unfriendly, it's business-like. Recruiters want you to give them more information than is in your interests, and you need to give them just as much information as is in your interests. Anything else you can give them that doesn't hurt you is being friendly, as is enjoying a bit of chit-chat, sharing a joke, talking about the weather, etc. Recruiters understand the game pretty well as they do it all the time, and will try it on, but aren't going to be offended if you are firm in refusing to prejudice your interests.