It's not a question of proper english, but of whether you consider it a complement - a pretty or beautiful man is something people sometimes use to describe new wave goth boys, skinny sculpted faces with arching eyebrows.. what is sometimes called "effeminate" but if you don't want to gender these things, then it's just the detailed, more curved, long-lined type of attractiveness. A pretty looking piece of pottery would also going be different from a handsome one, which would be nice looking, well-fitting and pleasant, so a handsome woman is good-looking but not breath-taking.

I think the confusion comes with context/use. They all very generally mean "good-looking" so can be used for a whole range of judgments (like, "shrug yeah he's handsome" or "oooh he is hand-some!"), but in direct comparison handsome is a vague positive, pretty implies decorated, and beautiful has a deeper connection to aesthetic pleasure.

You can use "handsome" to refer to a positive sum of money too, whereas "beautiful" is one of the three aspects of the world philosophers traditionally investigate (the true, the beautiful, and the good...). So it's not that the words are male or female - it's that we've applied words to men and women. Men are big, positive, strong in their good looks. Women are worthy of being gazed upon and pondered over as lovely mysteries