Of course you wouldn't know if someone did pass, but don't forget hands, shoulder/hip ratio, gait, adams apple, brow, voice etc. There's lots of ways to tell a persons sex.

I would say any that "pass" are most likely just falling into an uncanny valley where you think "Is that a man pretending to be a woman, or a reeeeeeeeeeeeaaaallly masculine women with a hormonal disorder? How do I avoid causing a scene?"

I do find TIFs on testosterone are sometimes really hard to tell though, once their voice breaks, but again IRL shoulder/hip ratios and gait and pretty obvious.