I'm writing into an old shared universe in which the use of the words "male" and "female" are so absurdly stereotyped that it reads poorly in this day and age. In order to mitigate this, I have decided that in the context of the part of the world I'm building, these words indicate separate subspecies of humanity, not biological sex. Therefore, I have created the following:
Imagine a race very like humankind that has such a low degree of sexual dimorphism that males and females are indistinguishable from one another in day-to-day life. An individual's sex can be readily determined by even a cursory inspection of the genitals, but no other single physical attribute is a certain link to sex. The breasts have been internalized. Other physical characteristics have also been regulated and normalized to some degree, so most everyone has roughly the same skin tones, eye colours, hair colour and texture, etc. Also, most everyone is pretty broad and husky; the hips are about the same shape for everyone, and pregnancy isn't very obvious except in the very latest part of the term, and even then is only readily detectable when an examinee is unclothed.
A few generations ago, this population was isolated and genetically re-engineered from human stock to be this way, so I already have a plausible backstory justifying how this came to pass. This state of affairs isn't a shock to any of these people; they were isolated when they were re-engineered, so they never knew what it was like to be able to differentiate between the sexes and it doesn't strike them as at all odd.
Their environment, to which they are physically constrained, is rural and temperate, with very little seasonal variation (I'm aware that I need seasons for many kinds of agriculture, but I'm not going to sweat that). This requires that fairly heavy clothing be worn year-round. Their society is agrarian with touches of hunter-gatherer, and very loosely based on the dark ages in terms of technology. Importantly, I have already decided that an individual's sex is a subject of deep privacy, and speculating with any degree of publicity about someone's sex is one of the deepest taboos.
Of course, this has had an enormous impact on their society. Sexual discrimination is all but non-existent (instead, people tend to discriminate on the basis of sub-species, but that's another story). Social roles that we might recognize as being gender roles are apparent (eg. that person goes out to work the fields, and their partner instructs the children), but are for convenience only and are largely not tied to biological sex. Language has evolved into a gender-neutral dialect; although a few ancient legends have survived which speak of male or female characteristics, it's generally understood that these stories are clearly written about gods or demi-humans, and are not to be compared with current society. A few individuals have dissented against this interpretation, but they have historically left this society to live on their own in the wilderness.
Given all this, what's the most important consequence of this change that I have overlooked?
EDIT: Genetic diversity isn't an issue. The process of the genetic re-engineering has eliminated genetic diseases. It bears mentioning, I suppose, that the story is only a few generations into their exile; long enough that the origins of their incarceration are now distant legends, but not long enough that the consequences of the small initial population have really started to show.
EDIT 2: I guess I wasn't explicit enough, my bad. The feeling for this subrace is one of extreme social conservatism. Think of the Puritans, perhaps, and the comparison with Pratchett's Discworld dwarfs is well-founded.