The height analogy always made the most sense to me. Like yes, if somebody is 6 ft tall, they are overwhelmingly likely to be male, but that doesn't mean we need to go schedule the entire WNBA for double mastectomies because "tall bodies are male bodies".

There's also the fact that if there were such a thing as "male brains" and "female brains", the occurrence of men with "female brains" and vice versa would disprove their existence, as they would no longer be "male brains" or "female brains" any more than 6 ft is a "male height".

Then there's the fact that even if somebody's brain did more closely match the opposite sex -- so what? The determination of sex, in terms of biology, which applies to wayyyyyy more organisms than just humans, relates to gamete size, nothing more, nothing less. Brains can have correlations with each sex but they are not any part of determining sex -- there are literally plants that are male or female, and they don't have brains at all. This is the same reason why the "sex is a spectrum" logic is flawed -- you can't be more female or less female: you might have more traits typically associated with producers of large immobile gametes, but having, say, a PCOS beard doesn't mean you're somehow "less female", it just means you have one body trait more commonly found in ejaculators.