@Cadence is entirely right in their assessment: there have been numerous cases throughout history where the expansion if an organized, agrarian society has been opposed by tight-knit groups of predators armed with human intelligence: indigenous peoples.
This isn’t just applicable to industrialized cultures practicing colonization across the world in places like Australia, Latin America, Africa, and northeast Asia (Russia and Siberia). It's also applicable to pre-Industrial Revolution civilizations like the medieval one in your OP. The Romans rather infamously crushed numerous Stone and Iron Age agriculturalists and hunter-gatherers in France, England, Germany, and other places in northern Europe. China spread north and south from their initial birthplace along the Yellow River and crushed and incorporated less organized peoples along the way. The only reason they weren’t able to crush Outer Mongolia is they had trouble dealing with the logistics of the steppe nomads. The Japanese destroyed the Ainu. The Inca crushed and incorporated a lot of less organized peoples to form their empire. There are a lot of defeated, less organized cultures that show up in almost every culture's histories as footnotes. The list goes on and on and on. It's a recurring theme in human history.
The closest thing you're looking for to your setting in historical terms is the settlement of the American West and the displacement of the many, many Native American nations that lived there. Many Native American nations at the time have traits that resemble your species: they didn't have permanent settlements or build cities, they lived as semi-migratory hunter-gatherers, and they didn't have heavy industry (this varied, of course, for example many nations did build large forts, but the broader point is they weren't organized in the types of agrarian societies common in Africa, Mesoamerica, Europe, or Asia, especially in the Great Plains region). There's been a lot of debate as to how the Europeans managed to displace the Native Americans but in terms of micro-scale interaction what really won the West for European settlers was that there were a lot more of them and they just. Wouldn't. Stop. Coming. The Native Americans and settlers butchered and back-stabbed each other with alarming frequency but the problem was even after entire towns were burned to the ground and their inhabitants killed or enslaved more European settlers would simply come from the east to replace them, whereas the Native tribes couldn't sustain the heavy losses. Just read the diaries of some of these settlers, even though they knew settlers in the area had been killed by the natives less than a winter before they were still rosy-eyed and optimistic about trying to settle in such an area. Many believed they were protected by God, or that the danger of Native Americans was overblown, or that European-style civiliztion would inevitably triumph, and were completely oblivious to the danger.
The only thing historically that could have preserved the Native American tribes would be them giving the Europeans enough of a bloody nose that they decided moving west wasn't worth it and force them to respect some arbitrarily set border. This almost happened when Roanoke Island went bust and Jamestown was in the toilet, England almost gave up on colonizing the New World until John Smith managed to turn the colony around. However, even this might not be enough. One of the grievances behind the American Revolution was that King George III banned settlement west of the Appalachians because they were afraid of the colonists aggravating the natives and starting a major war, especially since one of their neighbors was the very organized Iroquois (i.e., Haudenosaunee) Confederacy. The colonists ignored it and started going west anyway.
Indeed, it's even been suggested that one of the reasons agriculturalists won out over hunter-gatherers despite hunting and gathering being a much healthier lifestyle that offered more leisure time and less overall stress than primitive agriculture is that agriculture allowed a plot of land to support more humans in a smaller amount of space, and these miserable, disease-ridden humans were too organized and too numerous for any hunter-gatherer band to stop.
It should be noted this isn't too different from what humans did as hunter-gatherers, spreading out into the wilderness and killing off threatening megafauna in opportunistic encounters as they went. In some places like Africa or Siberia the megafauna were too hard to get at, but all that meant was they started to go extinct when humanity started to get better weapons.
This is what you're looking at with your story. Wave after wave of colonists setting out again and again to try to settle the wilderness, only for most of them to get butchered by the local wildlife. Then another batch try again, believing that unlike previous efforts theirs are blessed by God, or in search of wealth and riches, or out of desperation due to being forced out of their former cities by political disputes, famine, poverty, or lack of opportunities. Most die. But a few of them survive. And with that, they slowly chip away at the wilderness, bit by bit, until they've swallowed it into their domain, and the human empire expands across the continent.
On top of this, your premise has basically thrown out diplomacy as a possible alternative. It's not clear if the megafauna can even communicate with humans, and even if they could it's no guarantee the humans would recognize them as sapient. Heck, even today humans recognizing the sapience of great apes, cetaceans, parrots, corvids, and elephants and whether our treatment of them is ethical is controversial. Not to mention a common theme throughout human history is that the more organized invading force often saw the native population as subhuman anyway and treated them as such (and again, this isn't just a colonization thing, the Romans saw the Germanic peoples as little better than talking animals or savages. Part of the reason people like the Goths winning against Rome took the Romans by surprise is the Romans had been so brainwashed by propaganda that the "barbarians" were uncivilized idiots they ignored the fact that the barbarians had become organized and were politically savvy). Even if they can communicate, it's almost a guarantee that the people actually interacting with them won't want to. In cases of colonial warfare with natives, it's almost invariably people on the frontier who have lost loved ones to the conflict that least want peace. For example in many New World countries the central government tried to preserve peace with the natives but the settlers were the ones least sympathetic to the poor treatment of native groups and most likely to start violence with them.
The only, and I mean only, things that have ever stopped humans from colonizing an area are either environmental conditions that are too hostile for permanent settlement (and even then, that won't stop humans from trying anyway), or enough of a military response that forces humans to back off via threat of force. And the only way you could do that against a medieval human power is either a standing army or organisms that are simply so overpowered that humans can't survive against them. Of course, that raises the question of why your animals haven't wiped out humans yet. If a group of dragons is powerful enough to wipe out a human army, then why doesn't a displaced flock of dragons simply wipe out a town to make a new nesting ground. When humans kill humans in the long term there are still humans left at the end of the day. If megafauna wipe out humans too much humans go extinct.
Even in a situation where the whole ecosystem turns against humanity Pandora-style, what humans will just end up doing is burning the entire ecosystem to the ground and importing their own plants and livestock to replace it. This is basically what happened in New Zealand. New Zealand (and to a lesser degree Australia) was utterly unsuitable for European-style agriculture because it lacked a lot of the commensal organisms that made European-style agriculture thrive, such as dung beetles to break down and recycle ungulate poop and earthworms to break down and aerate the soil. What did settlers do? They ripped out the New Zealand ecosystem wholesale and replaced it with a mishmash of Eurasian species, creating a fake Eurasian ecosystem in New Zealand.