From what I've looked into ton this argument, I say there was some discrimination but I don't know if these groups were ever widely cast out of whiteness politically. The examples of, say Irish and Italians being placed in a non-white group were usually at the behest of several small companies. Here is also a good thread on this subject. https://twitter.com/la_chatte_verte/status/1600990191011139584

I'm not sure if what Ben Franklin said was serious or a sign that his views on racial identity were popular, but he did group only Europeans as "swarthy", while everyone else in Asia and Africa was "tawny".