**Humans are more social, we collect in larger groups.**

  This may very well be the reason humans won out over neanderthal, who were larger, stronger, and tougher. Human had no physical advantages over neanderthal.  But the average neanderthal tribe was around 10-30 individuals.  while human bands range from 30-150. These are both controlled by how big a certain part of the brain is(likely the neocortex), basically how many other people we can keep track of. This is not a function of reproductive rate but just how social your species is. Orcs may breed faster but if they attack in groups of 10-20 they will basically always lose to a human hunter gatherer band of a hundred or more. 

Strength is all well and good be it can't make up for nearly an order of magnitude more enemies. Once humans start forming villages and cities the difference just becomes more exaggerated. Dwarves need highly structured laws to collect in groups of a hundred while humans can do it instinctively. Elves just fail to collect in large groups, instead existing in separate tribes. Halflings and goblins max out instinctively to groups of 30-60, so closest to humans, which is why they are still around, but they don't have the numbers to compete with humans directly. Expect most of these races to exist in marginal environuts what cannot support large numbers of humans.