I've worked with many trans people. I actually found it easier to tell who was trans the more trans people I met.

If you can't tell men from women then you obviously have some kind of face blindness issue. As I say, babies can tell. Don't assume everyone else is as unperceptive as you. No, I've never had trouble telling which sex an androgynous women is. I actually think it's weird that's the example you give.

Safeguarding - men commit 98% of sexual assaults and transwomen offend at the same rate so they need to have different safeguarding rules than women. Only men can impregnate only women so in situations where sexual assault are more likely to take place men and women should be separated so if an assault does happen the victim isn't left with a baby as well. Maybe you're a sexist with and unconscious bias and you don't think gender non conforming women are real women?

Basic human dignity - people are more uncomfortable with people of the opposite sex seeing them naked than they are of their own sex. This is true across cultures and is likely instinctive. It's been consider part of treating people with dignity throughout human history, until now because apparently it's mean.