There's an angle to this that I think you could take which hasn't been addressed:
Instead of posturing why you don't do this because of the risk to OTHER clients or yourself...
Explain why you don't do this because of the risks to the client requesting it
Generally speaking, you always want to posture this in terms of the person requesting it, rather than yourself, and why the requestor "doesn't actually want this". Place the negatives, and the potential burdens, squarely on them and not yourself, and posture it as something you're not doing to better serve their interests in not doing it:
"If I installed this type of software for every client, there would be the risk of a breach in my computer, and they might be able to read your proprietary data and code, which would be a security risk to you. I value you as a client, and I understand your concerns, and that's why as a professional this is simply something I do not allow, as a general rule to all my clients, for your own safety."
Posture this as you doing them a favor and looking out for them by not doing this, ever, as a general rule. If they insist, offer them a solution that would make you comfortable but places all related costs on them, up to and including that they purchase related equipment for you, and they pay you for your time setting up the associated environment(s). If they're serious and have a serious concern, they will pay for it. If they're not (or if they're just being arbitrarily abusive) they'll move on to someone else, and you will be better off for it (since you've said you aren't desperate for this work).
Contractor vs Employee
Furthermore, another angle not covered here yet is that this degree of client involvement and interference into directing your work risks creating (in the US) a legally defined employee/employer relationship, rather than one of a contractor.
If you are working as an employee of a contracting agency, this is not an issue (you are already in an employee relationship with someone in relation to the job in particular, and if your employer wants to let someone else they're contracting with direct you this much that's within your employer's purview: but it's also an onus on them to provide equipment and be liable for creating any related security risks as your employer). If you are working as a fully independent contractor (which is what I assume in this case, from tags and wording), this is a situation your clients should wish to avoid: they are commissioning work from you, not hiring you as an employee, and that distinction is actually important.
Note that this is just a rough opinion, and if you have specific concerns you should hire a lawyer to look into specific situations (and also let me insert a generic "I am not a lawyer, nor am I your lawyer/etc" disclaimer).
I would personally recommend that you be careful with how much you share with them, and posture it as a matter of professionalism that you are simply logging billable hours in relation to work product, versus detailing your day. Focus on what you got done in relation to a given set of billable hours, rather than on logging breaks/etc as such. There's a degree of oversharing which can set up the wrong expectations.
This is someplace where when working as a fully independent contractor, I would, personally, take a page from how attorneys generally handle billable hours. Yes it includes things like "research" and the time spent doing that. No it does not (that I've ever seen or anywhere I've worked, having spent time in a law office) include things like "took a break". Document what you did, clock the time on it. That's what you bill the client. The rest is actually none of their business: they're paying for a result, they're not hiring you as an employee. (unless you come to a different mutual agreement to work as an employee for them)