There is little reason to assume that your hermaphrodite humans would not be stable.
It is possible that individuals might choose to take roles in society which are associated with stereotypical male or stereotypical female behavior in our society. After all, your society will need stereotypically male specialists like blacksmiths, lumberjacks, hunters, leaders and (if they are warlike) soldiers. But considering that there are no objective physical characteristics which clearly state that one belongs into a male or a female role, there is no inherent reason why any individual would be pressured into taking a specific role. There is also no reason why anyone would still think in categories like feminine behavior or masculine behavior.
It might be possible to introduce a caste system with some castes being expected to behave stereotypically male and other castes to behave stereotypically female. But there is no reason to expect that such a caste system would be a logical conclusion. You don't need one when you don't want one.
I'm afraid that stronger and more violent will specialize in being males, while weaker will specialize in being female,
If this is the case, then it would be reasonable to assume that the more masculine people feel sexually attracted to the more feminine people and vice versa. So the masculine and feminine genes would constantly crossbreed, leaving no chance for either trait to stabilize.
Those traits would only stabilize if people with these traits feel sexually attracted to people with the same traits (butches being into butches and femmes being into femmes). But the question is if such partnerships would provide an evolutionary advantage over the more heterogeneous partnerships where the partners complement each other or not. This creates some interesting questions which also have parallels to our society, namely the debate about adoption rights for homosexual couples. This might be an interesting concept to explore in detail in your work of fiction.