I mean sure out in the streets in most cases people will be polite. But when you see mainstream media portrayals of high school with the jocks and cool kids, as well as the losers and nerds. Those popularity levels, and those bullying, and that social ladder that occurs naturally, all of that originates in large part based on looks.

Also, while I'm polite to people that have a criminal like face, I'm sure there are many that wouldn't be.