The thing is, if you refuse to be called a "Nazi" or "racist", they'll keep insisting that you are one. I'm assuming this is as they believe you want to be dubbed the complete opposite or to not have such ideologies attached to your name, so my idea would be to just own it. But there's also the problem of assumptions they would make for people they label as such, even for the lightest displays.

Here's a related meme which helps illustrate my point: https://i.imgur.com/YAUHN7C.png