I agree with you, and I think most people do, but you're not going to find much agreement with radfems (I'm not a radfem). However, there is a problem with this line of thinking.

Someone like Blaire White or Nikita Dragun is going to essentially be seen as a woman to society. There are absolutely trans women who you'd never suspect until they told you. But how do you define that, especially in law? How do you know whether the trans person in the bathroom with you has a penis or not? Are you going to check? What if they have a vagina but are still 6'5 and 200 pounds?

Plus, there are plenty of women who aren't very feminine. What if a trans woman wants to wear boxers and short hair, likening themselves to a butch lesbian? There are lots of women who dress just like the average guy does. If a guy is wearing the same clothes, and declares himself a trans woman, the only ways to deny it are either saying women must wear feminine clothing or that trans women aren't women. That's the only way to be consistent about it.

Trans ideology is full of things like this, where you either have to be sexist or overly permissive. That's why we end up with so many generic phrases like "trans women are women" but a refusal to actually define anything or provide any coherent arguments. The left in general has decided to go for the overly permissive approach, because they're trying to be consistent and nice without having to own up to any of the sexism that is unfortunately inherent in trans ideology.