Well it is and it isn't. Genetics don't quite work like that, in that the genes for skin color will have other effects. Skin color is used successfully as a group indicator, and as that it's a veneer but other traits have evolved around it such that as an indicator it indicates far more than it alone would.

You are correct that judging peoples skin color could be looking at just the veneer. It's much the same as the social customs of people which are generally polite and nice. In either case getting past the veneer is necessary to find what is actually there. The problem though is in most situations getting past a veneer is really not possible, so you usually have to generalize on what you have known in the past.