Because for a certain segment of the population (and no, I'm not referring to the Jews here) the Nazi is the pinnacle of both absolute human evil and white supremacy. So they refer to anyone who isn't with them as a Nazi for a couple of reasons.

The first is the negative history of the label, and they wield it as a threat against the uncommitted or wavering supporters (only a Nazi would be against us and our policies, and you're not a Nazi, right). They also use that negative association in their effort to discredit and silence active opposition.

Then there's the white supremacy angle. White people (specifically) opposing their policies aimed at forcing equality and/or equity is clearly because of white supremacy, and Nazis were the biggest proponents of that, so white people opposing them must be Nazis.