Because we aren't queer enough "like a gay man should be" because the only thing gay about us is that we're gay. We aren't actually special, so we're basically straight to them

That's my impression, too. I also think that the "transbians" (who are of course running the show) can't abide gay men because:

• they don't wanna fuck ya, so what use are you, anyway?
• related to the above: not being female-attracted (unlike the L and the B), you can't provide that all-important VALIDATION = you're disposable.
• as dress-wearing dudes in a RuPaul's Drag Race world, they're TERRIFIED of being taken for drag queens themselves... and thus for gay men. Since that'd put an end to their precious "I'm a LAY-DEE!" fantasy. So you guys are collectively on their shit-list for producing the likes of Trixie Mattell.
• to them, "hating men" proves that they're Real Lesbians^TM , but naturally they don't dare direct that ire at STRAIGHT guys (who wouldn't stand for it, plus these "trans" dudes are of course straight themselves)... so you're the safe "smash-the-patriarchy!!!" option.