When you say "it can and has been measured", I've yet to see any of that stand up to scrutiny. You should read up on how these bogus studies have been shown to be worthless. But there are a couple of problems with trying to draw the conclusions they draw.

First, what are they comparing their populations to? If they are saying these brains of a trans population are more like women's than men's, they already have the sex based classifications set. They are not looking at their brain data and saying "hey, there's something wrong here, these seem to be women's brains", no they are arguing from the conclusion they want to find. And there are easy ways to show just looking at those same brains which came from males and which came from females. Talk about cherry picking data to serve the conclusion they were looking for all along.

Secondly: how does anything about these very debatable observations in some people's brains invalidate the simple fact that sex refers to our means of reproduction and truly all of these people in these studies are classifiable easily based on their reproductive function and that classification is easier and indisputable whereas the brain thing has very weak evidence? What does it mean to have an effeminate brain? How does that over-ride anything about what it means to be male or female? It doesn't even complicated it.