"men and women have different brains, as that becomes a slippery slope. But... is it the hormones? The reason I'm asking is because we seem to all agree that men are more violent, which is why we don't want them in our spaces"

On average, males and females do differ in at least 30 morphological brain features. That's not a slippery slope, that's just empirical data about measurable differences. At the same time, there is a lot of variation within each sex; there isn't really such a thing as a 'typical' male or female brain.

https://www.researchgate.net/publication/228549134_Gender_Differences_in_Human_Brain_A_Review#:~:text=Male%20brains%20are%20about%2010,more%20neurons%20to%20control%20them.

On average, healthy males who have gone through puberty have much higher levels of testosterone compared to a healthy females. Testosterone "activates the subcortical areas of the brain to produce aggression" so it's perhaps not a surprise that men tend to be more aggressive than females.

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3693622/#:~:text=Testosterone%20activates%20the%20subcortical%20areas,testosterone%20to%20reduce%20its%20effects.

There may be differences between brain structures that also account for differences in aggression, but I don't have any resources on that.

Please note: There are some horrible misinterpretations of the science to fit particular agendas. If you want to understand what science says about gender and sex, don't get it from a radfem source, as you'll most likely get a biased interpretation. I've even seen a lot of bias on blogs and articles that don't have an obvious agenda. If possible, read scientific reviews for a good overview and avoid science from popular sources.

Edit: There are differences, on average between male and female behaviour that have biological roots. Of course the environment also has an effect on behaviour and society can reinforce and enhance preexisting differences. The ideology about sex differences in behaviour being entirely down to socialisation is completely refuted by science. Sex differences in behaviour exist in other primates too, despite a comparative lack of sex-biased parental care and socialisation:

https://onlinelibrary.wiley.com/doi/full/10.1002/jnr.23862

Review of toy preferences in human infants: https://link.springer.com/article/10.1007/s10508-019-01624-7#Sec56