I would say that is not so problematic, the main issue is to find jobs/roles for men and women, that were strongly separated between them in the history of your planet and that had an important role in their lives (so it could affect their look).
The height is not so different so the thing you should focus on it the weight.
There are plenty of possibilities that could determine their weight and with weight strongly connected strength.
Especially the strength is ability traditionally useful for men (as women were at home taking care of children), so you just need the reason for men to fight and for women to not doing such a muscle activity.
(for example very safe caves all around the planet and less of vegetation so the women could hide, but they wouldn't be so helpful outside, and the men can be always outside trying to get something to eat.... but here I am making stuff up, there is a lot of other possibilities for you, like riding specific animals, defend some specific kind of homes or building them etc.)