'Crippling' inefficiency
I have a simple answer for you. Look at the Skaven from Warhammer. They're basically a social Darwinist society. They're a race of ratmen that can swarm the world with sheer numbers, are one of the most tech savvy races, adept mage users, capable of creating complex tunnel systems that allow their armies to strike from below and masters of chemical and biological warfare.
Skaven incursions are amongst the worst possible thing any faction in the Warhammer world has to deal with. Skaven have numbers, technology, magic,...by all means these guys should not only be winning, but curbstomping their opponents. They do curbstomp enemies in battle, but they don't win the wars.
Why? Simple, they're a bunch of backstabbers. How does a Skaven advance in society? He kills the guy above him. This of course doesn't matter much on the bottom ranks (slaves), but the moment say a soldier kills a commander a problem usually occurs. The soldier rat could have been a good soldier, but it doesn't mean he's a good commander. And usually the newly promoted commander rat gets killed before he can learn his new tasks by another soldier rat and the cycle begins anew.
Killing the guy above you and grabbing his position doesn't mean you're good at actually performing the duties that come with the new job. This leads to an overall loss of efficiency and competency. The Skaven are their own worst foe. They start out good, then they start killing each other and weaken their overall army and then the humans, elves,... win against the weakened army.
It's the same problem Stalin had with his new officer corps when he killed the old one off. The new guy doesn't have the same amount of experience as the veteran.
This society overall would have a serious issue with efficiency. It's not just the military. The new accountant could kill the CFO (Chief Finance Officer) of a company, but that doesn't mean he or she is going to be a good CFO.