Plenty of anti immigration rhetoric from the right is focused on latinos/mestizos. There are plenty of places where you can find right wingers bemoaning the ills of mass migration of mestizos - disappearing white neighborhoods, crime, drugs, loss of political power etc. Trumps anti immigration politics were hugely popular.

That said I do think there is a divide in the level of reactionary hatred directed towards blacks vs mestizos. No one cares about "white" latinos (in fact they are largely a boon to the right) and mestizos are generally more productive and less repulsive in character than blacks. They also have far less cultural, political, and social power than blacks - blacks in general take up a huge amount of mental space in modern white Americans minds while mestizos are largely a semi hidden underclass of proles. Blacks exercise far more importance in society than their position as a poor underclass suggests due to the liberal whites obsession with them while the mestizo situation more or less reflects usual class relations. Frankly you also see far less white women chasing mestizos compared to blacks so there is less of a visceral threat and reaction by white men.

Also blacks are also demographically replacing whites, just at a lesser rate. Usually people cite the 10-15% black population as remaining steady - in reality the black population has grown immensely, primarily from higher birth rates, it's just a trick of statistics that makes it seem constant due to the massive latino increase. In your grandparents day there were about 9 whites for every 1 black - now there are about 4 or 5 whites for every 1 black.