There are two questions here. The one you asked and the one that remains unasked.
Is it common? I don't know if it's common as such, but it's certainly not unheard of. It happens and I've known several occasions where it has happened.
The unasked question is "what would motivate a person to do this?" and the answer can help explain why it's not unheard of.
Assuming you're not leaving anything out, it appears to me that the boss is very concerned about sexual harassment complaints but also wants to maintain a certain "image" to the customers. Without a dress code or a uniform, he knows he can't mandate women wear dresses. That is at a minimum sexist and specifically saying that she should wear a skirt or dress would invite a sexual harassment complaint, especially since he's already indicated that he feels she looks like a man.
The reason why he would want her to quit is because he can't fire her without inviting that complaint. Firing someone is kind of like accusing them of something. That person can then defend themselves as to whether they were terminated for cause (at least for purposes of unemployment compensation, possibly more depending on the circumstances). When a person is charged with a crime, they get to plead not guilty and the state has to prove that they did deserve to be arrested and jailed. Quitting is like pleading guilty (in most cases). When you plead guilty, you forfeit your rights, including the right to appeal. When you quit (except in cases of a hostile workplace), you're usually losing any chance at unemployment compensation, etc.
I would bet that this boss would just fire your coworker if he could and just say "wearing a dress is the image we want to present because we want our women to look feminine and not masculine." But he can't. Instead, he has to hope that she quits.
The complaints about her work which seem inaccurate is also very common. It's called "building a book" where you establish ahead of time a supposed history of problems so if they're ever questions about it they can say "here's where I said she was screwing up, months ago". The problem for your coworker is that you really can't dispute it unless it immediately follows glowing reviews.
Here's the bottom line. Nobody wants to work where they're not wanted. Your friend probably should just start looking right now anyway. Additionally, I wouldn't quit either until I had something. But I'd definitely make certain I found something The writing is on the wall.
Of course, that's just my opinion. I could be wrong.
EDIT: Regarding the dress or her looks, my answer still remains. The guy apparently wants her to quit because letting her go has ramifications for him. And it's still sexist as hell.
TL:DR: It happens, but "common" isn't the word I would use. He wants her to quit to save trouble for himself since firing her would be sexist and possibly actionable. She should look for another job.