Let us suppose that males are much larger and stronger. That does happen in certain social mammals; for example Gorillas or Elephant Seals. This suggests that the male get large to compete with each other for mates and keep a harem.
Society would develop from hunter gatherer bands with one large male, and several to many smaller females. Smaller weaker males would be pushed out to roam on their own, or in nomadic bachelor tribes. Sooner or later, the strongest from a bachelor tribe, or perhaps a coalition of males would challenge the nearest chieftain/alpha/silverback and win, replacing him and taking his harem. Doesn't sound like the women have much choice.
But lets fast forward to the iron age. With a 20:1 ratio of females to males, and with the male's sole job being to fight and screw, the women will be doing all the farming, crafting, and inventing. Soon, their inventions will outpace the power of the single male. What if 10 women with iron spears, bows, and armor challenge a large male without? Sounds like the women will win.
The women could band together in tribe or villages of hundreds, and would chase off any incursion by nomadic males. They grew up without many males around, why would they want some around now? They would only keep as many men around as they decided were necessary for procreation. In this sort of society, I don't think it would even take until the firearms age until the women, by virtue of their technological progress, take full control of their society.
To me it seems like very large sexually competitive males is the perfect recipe for a matriarchal society.