Biologists came up the idea of defining of sex based on whether one's anatomy is meant to support either male or female gametes because it applies to all sexually-reproducing plants and animals. It's the simplest criterion for sex, one that can be applied across the board to all the diverse species that reproduce sexually, and allows for the fact that there is great variation in the exact ways sexually reproducing plant and animal species reproduce.

But that doesn't mean gametes or potential (future or past) for gametes constitutes the only fundamental criteria for defining sex in all species. In humans, the primary sex characteristics include more than the organs where gametes come from (the ovaries and testes).

This is especially true for female humans, because we conceive, carry and grow gestating offspring inside our bodies, birth them when they are already well advanced in development, then (can) breastfeed them. As opposed to, say, female birds that lay their eggs after they've been fertilized internally so their offspring do much of their development outside their mothers' bodies. Or as opposed to female fish who lay their eggs unfertilized, then the males fertilize them afterwards. And as is different to female marsupials, who conceive & gestate their offspring inside organs within their abdominal cavities, and give birth to them when they are very young and still hardly developed, then keep & suckle them in pouches on their bellies for the rest of their development.