Having different types of gametes is what defines sex. Without it there are no sexes. Species that don't produce gametes don't have sexes. Sexual dimorphism refers to differences in phenotype between sexes (i.e. individuals with of the same species that produce different type of gametes). Sex is indeed just the type of gametes an individual produces (or has the equipment to produce). Traits related to sex outside of the sexual organs themselves are secondary sexual characteristics. When talking about sexual dimorphism, we are only talking about secondary sexual characterizes, not the gametes or sexual organs. Sex itself is binary. Many secondary sexual traits are bimodal (i.e. breast size, height, adam's apple size, etc).

A good example of what no sexual dimorphism looks like in a sexual species is society finches. You cannot determine the sex of an individual by observing them or their behavior. The only way to determine the sex of an individual is to examine them to determine whether they have ovaries or testes (or do a DNA test). Extreme sexual dimorphism would be something like anglerfish. What we think of as anglerfish are all female, the males are tiny parasitic fish that merge with the female and become nothing but a pair testicles. Humans are obviously somewhere in between. We have clear sexual dimorphism, but it isn't extreme.