My experience is that your ethnicity can contribute to this if you are a minority, as well. Where I am, a lot of people say they find it hard to tell if an Asian person of androgynous presentation is male or female.

Are the people saying this also of Asian minority ethnicities? Or are they of non-Asian minority ethnicities? If they are of different ethnicities, can you share with us exactly what ethnicity they are? And what sex are they?

For various reasons, people of one race/ethnicity often show a reluctance to look carefully and familiarize themselves with the various ways that people of different ethnicities look. Which leads to the "they all look alike" phenomenon. This seems true amongst both minority and majority ethnic groups.

Also, across all races, women tend to be much better about telling the sex of other people than men are.