This is an interesting point.

But I do think whites self-segregate quite often.

Perhaps you are referring to legal requirements for hiring practices, which can limit the hiring of whites, in favor of non-whites. This is sometimes appropriately called 'positive discrimination' or 'reverse discrimination', and can unjustly favor a poorly qualified person and reject a much better qualified person who happens to be white.