Going by this argument, why even have gendered bathrooms? You're just blocking non-rapist men from entering, while according to them the rapist men can easily just walk in. Might as well make all the bathrooms gender neutral.

I've been invited by female friends to use the female bathroom, as a gay guy. I mean, it's nice they see me as a non-threat, but I don't wanna make things awkward for the other women that will use the same bathroom. Even then, it's not like I'm always 100% comfortable in male bathrooms. Growing up gay for me meant years of being uncomfortable of going to the male bathroom at school, having to share it with homophobic straight classmates. It's not nice being alone and vulnerable in a small space overhearing people next to you saying homophobic slurs.

I remember a popular transwoman influencer from my country telling the story of how once she had to block gay men from entering the female dressing room at the store she works. It was kinda funny to see her telling the story, and how she instantly assumed they were cis gay men, and instantly turned to blaming all cis gay men. They only interacted for a couple minutes and somehow she found out their gender identity and sexuality! What if they were non-binary, or bi, or even transmen? I love how they always say that transmen look exactly like cis men, but whenever something "problematic" happens they always know it's a cis person. They might not admit it, but they know that in the huge majority of cases it's easy to tell if someone is cis or not.

This comment isn't to say that bathrooms should all be gender neutral, or that gay men should be allowed in female bathrooms. It's just to point out the double standard in their arguments.