Transmen often use the facial and body hair they get from T as a tool to distract from and hide their female form/cosmetic surgeries. This is something Transwoman can't really do that(they more rely on clothes) and that might explain why transmen tend to pass better in general.

If you transmen are really so confident about your ability to pass, shave it off, like a lot of guys do. But you won't, because you know that unlike them, you can't pass as a man without it. And even with all the hair, there's still a heavy uncanny valley effect for even the most convincing of transmen.