Downvoting has always consistently been abused. Of course, so is upvoting. The difference is that, at the very least, critics will not be so harshly punished and punished with bad posting times to boot. That social stigma is also somewhat removed. Of course with these benefits, you've already noted the cost. The counter to this, I believe, would be allowing users to block those whose opinions are beyond ridiculous to them. Other than replying and pointing out just how ridiculous their opinions appear to be.

Think of it this way: If the only way to criticize somebody is to actually criticize them, isn't that better for everyone overall? Those who do not wish to waste time in criticism can simply eliminate those whose opinions they would never accept anyway, thus also removing any impetus for people to game the system to begin with. People who want a conversation can have one, and people who cry like petulant children or attempt to troll can more easily be ignored. I think this is the best of all worlds.