Actually, men are larger and more muscular than women for the same reason other male animals are bigger and stronger than females - the females invest a lot of resources into making bigger gametes and in mammals, in carrying the young and nursing them, so the males are the ones that physically compete with other males for access to females. Male polar bears are bigger than female polar bears because they fight each other during mating season and the stronger one gets to mate and spread his genes. Human males are basically built to fight other males (upper body strength in humans is way more sexually dimorphic than leg strength for example), that's why they're more violent and physically aggressive than women.

Now of course, violence as part of the culture is definitely a thing and culture in general has a giant influence on humans. So I don't disagree with what you're saying.