I always found the fact that dragons who possess superior qualities for leadership compared to humans didn't end up as overlords of humanity in many settings, or how cities were built, for that matter, when there were creatures out there who could easily annihilate squadrons. Anyway my question is how would humanoid races thrive in an environment where almost everything is stronger and smarter than them in a way that doesn't involve deus ex machina.
Well the simplest answer here is adaptability. Generally speaking you can find an animal that is much better at something than humans are, like speed, strength, not dying, etc. However no other animal can adapt, or failing that, change their environment like we can. Ants have superior leadership to most people, but ants aren’t the dominant species of life on earth. Our ability to manipulate our environment and the stubborn nature of humanity to spread and become more advanced over time means that, eventually, humans will gain weapons that can kill pretty much anything and everything, like what we have today. In any case, having your humans be the dominant race, even in a setting with dragons and such, wouldn’t strain the audience’s suspension of disbelief at all.
A reason for humans to still be around in a world that's generally speaking stronger than they are can be easily found and is still very much subject to your story.
Other life forms tend to not run around and murder everything for one.
Life forms don't run around murdering everything for fun.
That would be dumb (and a waste of energy). If they eat meat, they would kill of their source of food for example.
If they don't eat meat, they would endanger themselves to kill something that's no threat to them. If humans are not dangerous to the other species they don't really have the need to dispose of them.
Basically it boils down to live and let live.
Maybe they can even coexist. For example, a dragon could use humans to help him tend smaller wounds.
If it killed all humans helping, it would soon be alone and/or hunted.
However if he helped the humans to e.g. hunt and occationally drops some hunted animals as an exchange both parties would happily coexist.
Also you have to think about managing a kingdom. It's a lot of work. Maybe dragons don't want to lead loads of tiny humanoids that they can not really use for anything. It's a hassle to them. So they keep their distance or keep a good relationship. If they really want to stay alone they kill some that come to close and other people won't bother them anymore.
As for why humans build cities that could easily be destroyed by dragons:
Why would the dragon do that?
Conquering is no option. Cities are too small for them anyway.
A city itself has no value for a dragon. It's no real threat either.
Unless people actively hunt dragons that is.
Generally speaking you could keep the relationship between the powerful creatures and the humans this way.
Fighting each other yields no rewards for either side. So they don't fight.
Humans hunting a dragon yields a dead dragon. What for though?
If it was passive towards them there was no need to loose lots of men fighting it in the first place.
Same goes for dragons. Hunting humans may work as food. But humans are social.
If you eat a village, the rest gets angry.
They band together and next time shoot arrows on sight.
The time after that they developed a balista.
And the next time the balista head is strengthend with steel.
Eventually your scales give in and you die. For a meal?
Why not hunt some stags in the wood then? Easier to come by, not as social, can't fight back.
You can make this argument for every race really. Just make them focus on stuff the others don't want.
Here some examples:
Dragons like to live in caves on mountains.
Humans hate those because climbing is a pain, no crops grow up there and hunting in the forrest at the mountains feed requires you to carry everything back up. And water.... Such a hassle.
Humans like open plains. Growing crops, having some cattle, easy access to water.
Elves like forests. Their food grows for example on trees. Mushrooms or stuff like that.
Dwarves love metal and caves. So they dig into the underground.
Treants like to live in forests, maybe mushrooms grow on them as a parasite and elves tend to them, collecting those mushrooms as food. They peacefully coexist.
Gryphons are usually happy in the sky. They sit on mountains and hunt in the plains in the valley. Occationally they catch a human. But hunting them up those mountains is a pain. So humans learn to shoo them away with fire instead of fighting them...
And so on.
If there is no reward in hunting/killing the other creature, why go the dangerous (and potentially deadly) way of hunting it in the first place?
Same way rats do. By being able to survive in the margins, ignored as being unimportant, and an ability and willingness to survive nearly any environment or situation.
An elf would be hard pressed to scarf down a half rotten carcass in a Swamp. A dwarf will constantly lament his lost mountain home as he risks heat stroke in a desert. Orcs will fight amongst themselves and consider a pincer movement the height of warfare. And dragons don't co-operate with each other or take centuries to subtly take control.
To survive, humans as a race, not individuals, will accept humiliation in place of death, will lie cheat and steal to get the necessities of life, will sacrifice themselves for the group, will make a deal and actually stick to it, will use every dirty trick in the book to make the other guy die, and will work with anyone or thing that will let them survive and thrive for just a little longer.
And when everyone thinks that the humans are unimportant savages, who can be ignored or placated with table scraps, we will pull out our stolen elven and dwarven weapons, and come swarming out of the badlands with whatever allies we have, no matter how barbaric they are, and burn our enemies cities to the ground and place their heads on spikes.
If that doesn't work, we will work as obedient slaves, making ourselves invaluable to our masters, until in what may seem like no time at all the overlords discover that we are making the important decisions and they can't survive without us. when humans realize this, it isn't long before positions will change.
They may be smarter, stronger and better, but as people have proven against each other, we will burn our weaknesses out of our society when necessary and do whatever it takes to ensure we are the last ones standing.
The only way to beat humans is to kill as many as possible and ensure we stay small in number, or kill all of us.
It's much easier and safer to keep any relationship as a live and let live situation. Over awe humans sometimes, make sure they realize that it's safer to keep out of the way or even to be friends, but dont push them to the point where they feel they have to kill your entire race.
Dragons are few, humans are many. There's power in numbers!
Tech/magic can level the field and give the humanoids an edge.
Why would a dragon be interested in puny humans, as long as they don't bother it?
Dragons hibernate. They only get up to eat your cows and burn your city every couple of centuries. Don't be a cry baby.
Dragons are [insert limitation here]. Basically, for a balanced story, and one that you want your readers/viewers to like, the super monsters can't be all that super