(For clarity, the question has been significantly edited from its original version, both before and after I wrote the reply below.)
We've been assigned to work with a man who considers themselves to be
a woman.
Polite terminology here is "a transgender woman".
She's making some weird demands, such as getting upset whenever we use
the pronoun "he" (which we obviously do out of habit, I mean, what are
we supposed to do, change our instincts over night?)
Speaking from experience (plenty of trans friends, have screwed up their pronouns more than once): trans people are usually understanding of and patient with people who are making a bona fide effort to adapt to a change in pronouns. I've never had anybody jump down my throat for accidentally using the wrong pronouns.
The key words there are "make an effort" and "accidentally". If it looks like you're not trying, that's a whole different interaction, and getting people's pronouns wrong will give offense.
From the original version of your post, it's very clear that you do not respect your transgender co-worker, you do not accept her as female, and that calling her "he" is not an accidental slip but a deliberate choice. Claiming it simply as a matter of "habit" is disingenuous; small wonder that she isn't buying it.
and she also demands to use the ladies toilet, which makes our only
female coworker quite uncomfortable.
She makes weird noises in the
bathroom, she says, and insists on making weird, unnatural chitchat
with her about "the boys" (i.e., us...).
How to deal with this situation? We do not want to be discriminatory
against an LGBT person,
For the sake of discussion, I will suppose that you are genuine in this last statement. So...
If you had a female co-worker who was not transgender, and another woman complained that she was making "weird noises in the bathroom" and "weird, unnatural chitchat", how would you handle that situation? There are several possible answers, but I doubt anybody would suggest "stop her from using from the ladies' room".
Small-talk is a very cultural thing - e.g. Germany and USA have very different norms about chit-chat, and what's normal to a talkative American might feel "weird and unnatural" to a German. (Or indeed to an autistic person like myself.) But it would be ludicrous to escalate those cultural differences to "ban Chad from the bathroom" levels of drama.
If it's helpful to your other female co-worker, she might think of her transgender colleague as somebody who has been raised and socialised in one culture, has moved to another culture with quite different norms, and is doing her best to fit in but is still learning the ropes. She is probably aware that women tend to talk more in bathrooms than men do, and not yet sure on the difference between "normal female level" and "too much".
(Trans women invariably get slammed either for "not behaving like a real woman", or for "behaving like an exaggerated female stereotype"; there's no middle ground.)
So, handle the situation in the same way you would if it was any other cultural difference. If you choose to handle it differently and more harshly because she's transgender, well, that's pretty much the definition of "discriminatory against a LGBT person".
Obviously taking this to HR is extremely delicate, as their first
impression may very well be that we just don't like LGBT people.
That is... certainly a possibility.
I do agree with other posters that HR should, if nothing else, establish clear policies on who gets to use which bathrooms. But you are probably not the appropriate person to be raising that issue with HR. At most, you could let them know that there is an issue - "hi, we have a transgender employee, it would be helpful if you could give some guidance around bathroom use". Anything beyond that is likely to be unhelpful for all concerned.
Beyond that, leave the women's bathroom to those who use it, and address all your co-workers by the name and pronouns they've requested, both in their presence and behind their backs.
