You put 100 men and 1 woman in an island and return 1 year later and the population will have increased by one.

You put 100 women and 1 man in an island and return 1 year later and the population will have increased by 100.

The 1 woman on 1st island will try to pick the best man out of the 100 in order to mate, that way her 1 child will have the best genetics possible.

The 1 man on on the 2nd island doesn't care at all about how good his partner looks. He can impregnate multiple women, hence he'll breed everyone and spread his genes as much as he can.

Males have evolved to be mostly indiscriminate in their mate selection because, in most species, most of the biological investment in generating offspring comes from the female side. Therefore, for a man, any woman who might want to mate with him is potentially interesting, hence men find most women to be sufficiently attractive. Women, on the other hand, have evolved to be exceptionally picky on their choices of mates due to the fact they almost never experience scarcity of prospective partners, due to the inherent bias towards males being far more interested in them than the other way around.