Differences in group dynamics
One deciding factor is the psychology of the inhabitants. Human cultural similarities and cultural distinctiveness are shaped by factors such as individuals' competing drives to fit in vs. stand out, or our attraction to romance and reproduction outside the kin group. Those strongly influence things like migration patterns and new cultures splitting off from old ones. Or, imagine that our neural hardware for language were more hard-wired, such that everyone on earth spoke mutually intelligible languages.
As a third example, research has shown that humans have an almost universal tendency to believe that there are intrinsically different types of people -- and it operates similarly whether the groups are based on biology (race), kinship (ethnicity), or religion / purity (caste). Without that, humans might be much less likely to fracture into political / racial / religious groups and might have a more homogeneous global culture.
Other differences in psychology
Alternately, imagine a species that is less flexible than humans. Perhaps they evolved in a more stable, unchanging environment, and didn't develop as wide a repertoire of behaviors. Or, perhaps they had stronger and different pressures than we did, and so evolved a psychology that's more strongly oriented toward one thing. e.g., on a planet where dragons periodically swoop down from the sky, all people in all cultures might be what we would perceive as a "warrior race" (or they might be the Planet of Computer Programmers Who Never Ever Go Outside).
Differences in perspective
In either of those cases, you can imagine that those species might perceive individuals as being different in all sorts of ways that aren't apparent to us -- they'd only be a planet of hats from our point of view.
You can also imagine aliens coming to earth and being surprised by how all of us are so uniformly obsessed with language or building things and consider us a cartoonish planet of hats.
So, I would say that a planet of hats is not an unreasonable concept. But, if you deal with its inhabitants in any depth, you'd need to think about what kind of people they are.