This is just an observation, but it all comes down to perception/stigma.

A man like Jared Taylor doesn't have any biases that are inherently right-wing or left-wing. He only understands the importance of race, and that it's a much bigger governing policy than raising taxes or defending gun ownership.

Yet despite this, the media still slanders or forces on the label that he must be a "white supremacist" or a "nazi". Or anyone else alt-right must also be some kind of Walmart shooter extremist, even if they oppose violence at every level.