Many fantasy worlds have multiple humanoid species (elves, orcs, humans, etc) living in the same world, in addition to that, they often have the ability to interbreed. And I can't help but question their believability...
Well, first, magic is scientifically flawed! So if there is magic involved, then the scenario is scientifically flawed.
But if there's no magic involved, then there's nothing intrinsically impossible about having multiple distinctive, interfertile humanoid races. Two points to support this:
Our own pre-history had a time when there were at least three humanoid species on Earth: Mainline H. Sap, the Neanderthals and the Denisovians. There is good (but not yet certain) evidence that we have a few percent Neanderthal and Denisovian genes, so it appears that our own ancestry demonstrates the possibility of distinct, interbreeding humans species/races/types.
Secondly, dogs (and other domesticated animals) demonstrate that it is possible to artificially breed dogs which are all interfertile, but have distinctly differing body types. There's no reason I know of that something similar couldn't be true of people. Probably the most plausible way for this to happen would be for a single humanoid species to be split geographically and to evolve under different conditions to have different body types, and for the barriers to later be lost (they just end, or people learn to sail around them, or technology improves enough to cross the mountain range or something like that.)
It still seems pretty unlikely without outside intervention, though: Dogs didn't develop all their various forms under natural selection!
All fantasy races are different ethnicities of the same species:
Biologically and historically it makes more sense to think of the fantasy races as ethnic groups.
First is the historical portion.
The idea of the original creators of the fantasy races (Tolkien) was that the races represent ethnicities, not species. Recall that in Tolkien's book orcs were not a different species, they were fallen and corrupted elves. So even in Tolkien's books orcs and elves were not different species. I think the race of "Men" also finds its origin in Elves that chose not to be immortal. So in the source document of this concept, these groups were not different species. My understanding is that Tolkien is the originator of the fantasy races.
The fantasy races are not different species, they are different ethnicities/cultures. When the fantasy races were invented in Tolkien's books, human ethnicities were still considered to be considerably different because of lack of exposure to different cultures. So I think the fantasy races really just represent people from different regions (with cultural and some minor genetic differences).
To exemplify how most of the differences can be explained away from the perspective of ethnicity, lets go through a few. In Classic Dnd Humans are Europeans, orcs are savages of some uncivilized land/heathens, drow are similar to people of Africans descent (different in skin color only, the content of their character has been attributed to culture in Dnd), elf can be east Asians (slightly more petite and classically thought of as more philosophical because of Confucius), etc.
In this context the classical races we see in settings like DnD are just that, races. Race as a term has fallen out of favor, but the modern equivalent would be something like ethnicity/culture. And recall that members of the same species can look quite different, use dog breeds as an extreme example. So different cultures and minor genetic differences can attribute for a wide range of differences. Remember that Europeans thought for a long time that they were smarter, when really they were just better educated at the time, as an example of the impact of culture.
The biological portion:
I think taken in this lense the fantasy races make a lot more sense. After all they all have similar cognitive and athletic ability, which would not be the case in different species. They can all interbreed, which also makes little sense from a species point of view. However wildly different superficial features and lifestyles are perfectly plausible within a single species. So most likely the fantasy races are ethincities & cultures.
Is it scientifically plausible? Yes — From a certain point of view
Humanity has competed against a fair number of competing hominid species:
Why did our close relatives — from Neanderthals to their recently discovered cousins, the Denisovans, to the hobbit people of Indonesia — die out while we became a global success?
That is the million-dollar question. My view is that great variability in our ancestral environment was the big challenge of human evolution. The key was the ability to respond to those changes. We are probably the most adaptable mammal that has ever evolved on earth. Just look at all the places we can live and the way we seek out novel places to explore, such as space. (Source)
In other words, from a certain point of view, we had this very situation on Earth. Competing, related species all living at the same time. Why didn't the others, like the neanderthals, survive? For a lot of reasons that are entirely ignored in fantasy (and that's OK).
The problem with fantasy is that the various species are presented as intrinsically different. Orcs are piggish in nature with (in some cases) tusks, thick skin, etc. Elves are presented as magical (not exactly scientific...). Etc. In many instances authors are forced to simply "declare it so" to make these very different species work (and play) together.
Scientifically, there would be no hope or chance that the species could interbreed. There would be diseases that hurt one but not the others ... and occasionally diseases that jumped from one to another with devastating results. Given the fundamental competitive nature of humans (a competitive nature born, I might add, from millions of years of fighting to survive. It's one of the many reasons for our basically aggresive nature), we would have banded together to kill all the Orcs a long time ago. And we would have won unless they bred like rabbits.
So, there's scientific evidence that several similar hominid species can survive at the same time. But it's that similarity that makes it possible. That, and a lot of Darwinian luck.
It's basicly a question of evolution and genetics. The closest living species to humans is the monkey, but there were several human sub species like Neandertals and Denisovans.
While it's true that humans cannot interbreed with monkeys, Homo Sapiens did interbreed with neandertals and co-existed with them for a while. So if elves were genetically close enough to humans to be a sub species, they could coexist and interbreed as well. From the popular depictions of dwarves, they might be genetically even closer to humans than elves.
How did they develop?
The myth of small, stocky humans living and working undergroud is ancient, probably as old as the bronce or iron age. It stems from the fact that digging a tunnel into a mountain is extremely hard work, so you dig it as small as posible (crawling is enough). Children had to support their parents from an early age on, so they were sent into the tunnels to work. This impeded their physical development and they became dwarf-like.
Of course this is no distinct species, but it could be the beginning of one.
Can they live side by side?
The number and very different types of fantasy species is indeed a problem. A recurring theme in many fantasy worlds is racism and ecological competition between species.
- Humans are often depicted as "the young race", a new player that spawned in some far away location and spread rapidly over the whole world, thereby displacing the "older races".
- Hobbits / Halflings are generally integrated into human society. They are basicly small humans and are comaprable to modern human "races". Sooner or later they would be assimilated into humans and no longer exist as distinct race.
- Dwarves usually life under the surface of the earth, a place where humans cannot and don't want to life permanently. They found a comfortable niche to coexist with humans for a prolonged time.
- Elves usually live in old and wild woods, the places that humans have not urbanized yet. Either their woods defend themselves (magic) or the elves actively fight off humans who want to take over their land. Their only way to coexist with humans is to constantly fight for their lands.
- Orcs are just everyones enemy. They are also so different from humans that they're probably a distinct species and shouldn't be able to interbreed with humans.
Short Answer: no, it's not flawed, because of technological standstill, potential of quantifying magic into physics equations and evolutionary niches.
Long answer: first, many fantasy stories see a technological standstill. They dont develop beyond canons and musket like weapons if they reach gunpowder at all. This could be explained in part due to the environment. With more resources going into keeping your dead from turning to zombies or having to clear out some kind of cow-sized bugs from your neighbourhood can take a lot of your time and keep population down, stifling research. On the other hand strife like WWI and WWII saw a technological boom, so technological progress could be halted due to lack of resources and certain physics not existing or changing things. After all if a Dragon or cow-sized bug can live without collapsing under it's own weight, certain materials or physics must be available to allow/disallow certain things that are possible in our universe.
Second, magic often hints that it be quantified. Such as a mage only being able to transfer heat rather than create it, creating a temporary physics alteration to local space to allow it to happen. Think of it as quantum mechanics on a largr scale: when observed by a mage who uses a certain amount of (dark) energy, often referred to as mana, you change the outcome.
Thirdly, because the aforementioned technological standstill and the need to adapt more rather than have technology take over, sentient species are more likely to be diverse. Theres a group of people in our world that has lived so long in houses on the water of a shore that they are slowly evolving to be more aquatic than normal humans, and people in the himalayas have evolved and are still evolving traits to better deal with the lower oxygen and harsh conditions. So without technology like oxygen tanks or whatever and given a long enough time, different species could evolve.
Orcs could simply be a reaction to an extremely violent area, where murdering things randomly is more likely to keep you alive than spending your energy only to hunt. Elves might have learned to avoid and eventually co-exist with whatever forest they ended up in, dwarves might have been forced underground because of the wildlife or dangerous foe's and being smaller and sturdier helped them survive in their caves. This way everything can be scientifically presentable, as long as the reasons as to why it all came to be are there.