You would need an enormous pressure.
In normal primates, it's better for a large primate to be male, because males have larger maximum numbers of offspring, and size helps it be dominant and impress the girls -- and it's better for a small primate to be female, because males also are more likely to have few or no offspring, being cut out by the larger males.
Every bit of food that goes to making a female larger is one that doesn't go to making another offspring. That females must invest much more in their offspring than males do means this is much more vital for them.
This is not a barrier as such, but it is an obstacle.
Perhaps you could have oases of plentiful food, but males must leave their childhood oasis for an adult one, and the long treks between them make large size unfeasible. Meanwhile, in the oases, a female's biggest pressure is other females, and so large size is necessary.
Edit: hmm -- then the male could travel while young and grow after. Perhaps the females would eject the males whenever there was a shortage, meaning a male always had to be ready to make the starvation-rations trek.
Edit 2: To eject the males, the females would have to be already larger or work in concert, but if they pull it off, those that eject males by size would be better off. Larger males are a larger burden on the food supply. Furthermore, if they are harder to fight, they are easier to corner; a smaller male could, say, climb a tree to boughs too small for the females to follow, and perhaps even forage for food there, thus further decreasing its impact on the practical food supply. Keeping around the smallest males would ensure there were some for when breeding became practical again at minimum cost.