I think the flaw (for lack of better word, sorry) in your theory is that any female typical feature on a man or male typical feature on a woman is the opposite of the expectation. And because it’s the opposite, they don’t elicit the same reactions or responses that they would on a person who is the sex typically associated with the feature(s). Does that make sense? Meaning, it’s the very fact that they aren’t the sex typical of whatever feature that makes their experience different. A man with “moobs” is gonna be mocked or considered overweight, a woman with big breasts is often considered a positive thing in society (and the point where a man has “moobs” is still not comparable to the point where a woman is considered as having big breasts, if that makes sense?). Conversely, a woman with a “flat chest” is often mocked or deemed less desirable (ex: the “itty bitty titty committee” thing 🙄).

People are held to the physical standards of their sex, so having a feature normally associated with the opposite sex makes them an outlier, rather than giving them a kinship to the opposite sex- because those features would be unremarkable and likely go unnoticed when they are on a person of the sex they are attributed to. Kevin Hart, for example, he’s the average height for women. But since he’s a man, his height is always a punchline. Look at the height of basketball players, to use that example again- female basketball players average in height somewhere between 5’6-5’10. Male basketball players are often well over 6ft. Allen Iverson is 6ft and was considered a short player (there are shorter players too, to be fair).

It’s the very fact that they aren’t the sex associated with whatever feature that isolates their experience and perception (rather than connecting them with the opposite sex), as well as how they are perceived by others. So a woman with a typically male feature is going to be considered and possibly treated as “different”, where it’s typical and unremarkable for a man, and vice versa. And again, it’s worth noting that the point where it becomes atypical on the opposite sex than the expected is always going to be a point where it’s standard on the associated sex.