TLDR: best case scenario: de facto caste system; more likely: population replacement
De facto caste system
Already in past it was noticed that some ethnic groups seems to excel at particular tasks. For example Normans (Vikings) were much taller, stronger and fierce enough to be main source of mercenaries for centuries, hired even by not so close Byzantium. They managed to conquer for themselves multiple lands, from less known - Sicily or Kiev Rus. Their era ended because they simply integrated in to local population.
Humans as they are already diverse enough to have some groups over represented in specific jobs. For example: people who evolved lactose tolerance, have clearly had an edge in being shepherds. If we assume that there are some genetic differences in average IQ between ethnic groups, then we could reasonably explain why ex. Askenazi Jews are highly over represented among Noble prize winners.
Nevertheless, here we have people that normally mix, so if whatever alleles are highly beneficial they would spread so ethnic groups living together would mix. Assuming that's not workable with different species, you end up with peoples with absolutely diverse predispositions, that would naturally pick the best jobs for them, which would lead to job separation. Moreover, as in premodern society people team up in clans, then at best they end up separate but equal, or more likely create some kind of caste system. As caste system worked in stable way in India for millenia, I'd say that's the way to go. The only problem that I see even peaceful situation is whether all species reproduce in roughly the same speed, otherwise in long run, some would be outbread.
In recent years we started to get genetic studies of our distant past. Population replacement were highly common, even though we are one specie. In Europe, first neolithic hunter gatherers were partially replaced by Anatolian farmers. Later, in the most brutal way possible (in most cases, except maybe nowadays Spain, genetic data show that not even women were spared) those were mostly replaced by Yamnaya.
Similar process were happening already in quite recent times with Bantu expansion in Africa, which doomed most of Pygmy and Khoisan.
If we talk about more distant past - all what's left out Neanderthals and Denisovians is some tiny genetic admixture in out-of Africa homo sapiens. So I see the pattern, and it's not optimistic one.
As side note - please let me ruin story that ancient and premodern people were not racist in modern meaning of this term. Sort of true, just let me remind the fine print
- in Ancient Greece and Rome there was indeed equal opportunity... slavery. People, regardless of their race could be captured in war, sold as slaves by their parents (this part for sure in Ancient Rome) or auctioned to slavery to pay off their debts.
- As far as their knowledge allowed Greeks were trying to pioneer racism - considered "Asians" (Persia and surroundings) as civilised but spineless people who are easy to subjugate by any tyrant, while Europeans were fierce free people, hard to subjugate and even harder to civilise (Presumably I'm supposed to find this highly offensive :D ), while only Greeks were able to both be good warriors and good at creating civilisation.
Those who live in a cold climate and in Europe are full of spirit, but wanting in intelligence and skill; and therefore they retain comparative freedom, but have no political organization, and are incapable of ruling over others. Whereas the natives of Asia are intelligent and inventive, but they are wanting in spirit, and therefore they are always in a state of subjection and slavery. But the Hellenic race, which is situated between them, is likewise intermediate in character, being high-spirited and also intelligent.
Politics by Aristitotle, book VII, Part VII
- Tomb paintings in Ancient Egypt clearly show that Egyptians not only noticed, but even were even overemphasising phenotypical ethnic differences while painting Egyptian, Greek, Nubian and Persian
Muslims were slave raiding both Africa and Europe. I'm not sure whether it counts as racism, but white slaves had higher market price.
Already people, who were by all modern standards the same ethnic group, were willing to create convoluted stories to claim otherwise. Examples: Rome was claimed to be established by some Troy refugees. Later, in Polish-Lithuanian Commonwealth nobility was claiming not being related in any way to peasants, but deriving its origin from steppe Sarmatians.
In one of early contacts between Europeans and Japanese, Europeans had hired a Chinese scholar as translator. According to Japanese recordings he described Europeans as not much better than beasts, as they were absolutely unaware of even basic Confucianism. (to be fair: this part was culture based)
Racism, as understood nowadays, indeed came later, when superstition and ignorance were replaced by actual knowledge gathered in age of geographic discoveries and age of... reason. Then indeed humans were classified in to races that roughly match nowadays studies concerning genetic clustering.