Because it's an issue that only people who are, avoiding the word extreme, shall we say, deeply interested in the issue care about. Be that in favor or against, naturally this includes extremists.

Feminist and other hardline socially progressive type groups will attract all the weirdo outcasts and other people with unusual ideas including extremists due to tolerance of them and slowly those with intolerant ideologies will gain a foothold and there will be an internal split if you will between the rabidly passionate yet intolerant minority that demands everyone go there way while the majority increasingly gets alienated as it moves past the fringes due to the purity spiral.

The right naturally will be concerned with what the opposition is doing and will naturally see what the most extreme left is doing as the greater threat than the more reasonable moderate left who are willing to compromise on most issues. Naturally the far right has similar purity spirals and extreme ideas which in their own point of view are progressive in the direction they wish to progress in and the underlying social dynamics are very similar. So the fringes on both sides become aware to this fringe issue ad they are the only ones who care. Slowly these grow to more mainstream acceptance but initially only the most extreme right wing voices will get much coverage which tends to turn normal people off since they'll assume they hate trans people because of the bible or something like that and the more tolerant stance of "let people live their lives how they want" is more palatable (and I'd agree adult trans people should be free to make their own choices on this issue) but they've yet to get into the weeds of potential medical profit driven malpractice, the social contagion concerns, the potential of pedophiles to be using this as a shield to get to kids, the potential connections to mental illness as a co-morbidity, etc. As this issue progresses further into normality more people will become aware of these issues. I think we're already there but it depends on what circles you are in.

I think it's also fair to point out that a lot of average people are really stupid. America especially is known to have pretty piss poor public education. So many people may lack the sufficient scientific knowledge to counter the generally spiritualist trappings of gender theory.