I'd say you've got it exactly backwards. Sexual dimorphism is a term we use to talk about how different the males and females of a species are. For example, male gorillas are nearly twice the size of female gorillas, as opposed to humans, where the males are around 10% taller. So gorillas hav more sexual dimorphism than we do. Or think of birds, the different colors between males' feathers and females'. On the other extreme, I would have no idea how to tell the difference between a male housefly and a female housefly; they have much lower sexual dimorphism, although I'm sure scientists who study them could describe to me what differences there may be. Basically, if you take a male example and a female example from a given species, what differences will you see, in things like size, shape, color, amount or distribution of hair or feathers or whatever?

Obviously within one sex attributes will fall along some kind of normal distribution: there are taller men and shorter, more and less hairy, there are taller and shorter women, women with larger and smaller breasts, etc. But a shorter man is not more female than a taller man, because the dimorphic attributes are just a result of sex, not a cause of it. Which sex a creature is just means which role it is capable of playing in reproduction (supposing it's healthy and so forth). The external traits, which may be more or less dimorphic depending on species, might help you figure out what sex an individual is, but they do not determine it.

There's a kind of fish (sorry, I never remember the names of these things) where the males and females mostly look different and live separately, but a few males take on a female-looking form, a disguise, so to speak, and slip in with the female school. This strategy allows them to fertilize the females' eggs without having to compete with other males. In other words, they are still firmly within the male reproductive role (fertilize eggs) even though they appear female in terms of the superficial dimorphic attributes.

Most of the attributes mentioned in your comment, which the people you quote claim are part of the definition of sex, are not. Sex is which part of reproduction an individual can potentially provide. In humans this starts with which chromosome is provided by the sperm that fused with the egg to make you. The genes on that chromosome provide a blueprint according to which a body is built that in turn goes on to produce eggs or sperm. Egg-producing bodies and sperm producing bodies have different average size, shape, biochemistry, etc, which all fall under the category of sexual dimorphism, and while some of them may have some consequences for reproductive fitness (a woman whose hips are too narrow might have difficulty in labor, for example), they are not, themselves, of any importance in figuring out what sex someone is.