pretty much any exchange of labor (use of your body) for money is rape

No. Merriam Webster defines rape as: unlawful sexual activity and usually sexual intercourse carried out forcibly or under threat of injury against a person's will or with a person who is beneath a certain age or incapable of valid consent because of mental illness, mental deficiency, intoxication, unconsciousness, or deception

There are few jobs more dangerous, because of the risk of STDs, violence, and in the case of women prostituting themselves to men, pregnancy. Prostitution is inherently violent because, by definition, it involves unwanted sex. The victim not only has to bear the invasion of their boundaries and their most intimate places, but they also have to act like they're enjoying it. This is a form of psychological violence.

Your comparison between having a sugar daddy or working in a sweatshop doesn't work. If one were so disadvantaged they had to choose between sweat shop work and prostitution, the type of prostitution they would need to endure would be much worse than having one sugar daddy that you suck off MAYBE once a day. Such a person would likely have to work in a brothel and be raped several times a day, being unable to decline clients (though realistically, they'd like to decline them all). Having a sugar daddy that actually pays enough to live is extremely rare and not an option for most people. Sugar daddies do not represent the majority of people's experience with prostitution.