For many reasons.
The late Middle Ages are well within the Malthusian Trap. In the Malthusian Trap, population scales rapidly with food supply and other wealth.
An excess of food results in rapid population growth, which then makes food scarce again.
It wasn't until the Industrial Revolution that any part of the world escaped the Malthusian Trap; any civilization not in that Trap would not be "late Middle Ages" like.
So population ends up being concentrated where food is plentiful, and elsewhere starves. At best small colonies can exist elsewhere, with goods are sufficiently valuable people where the food is plentiful are willing to ship off food despite locals starving.
The Rocket Equation
Bulk transport is expensive. On land, doubly so.
A horse can eat 20 lbs of food per day. If the cart has a ton of goods (2000 lbs) that is 1% of the mass of the cart per day.
Moving at a human walking pace of 5 km/h (it is pulling a tonne), over a 8 hour day, that is 40 km for 1% of the mass of food, on high-quality roads.
Europe is about 5000 km from one side to another; that is 125 days. 1.0 - 0.99^125 means that you lose a good 72% of the food you want to transport in feeding the horses at best, and that ignores feeding the human or repairing or horse shoes.
Long distance transportation of food calories by land isn't plausible, even with high quality roads, prior to an energy source as abundant as coal to pay for the transportation work.
Water based transport is better; but mass cargo shipment in the late Middle Ages isn't something that plausible. You'd have to have ridiculous sized fleets and massive amounts of ship building to keep them intact.
The middle ages end with the 15th century; well into the 16th century oar-power was important. The Age of Sail is basically the time when long-ranged Sailing trade became dominant; as you want to be set in Late Middle Ages, you aren't in the Age of Sail.
Ocean transport is going to be better than horse-and-cart, but not so much so that you can practically have a fleet capable of carrying enough food for a non-trivial population over 1/4 of the world.
Starving is a lot of power. While the non-food producing areas could have a huge army to punish and invade the food-producing areas, this just moves the power to that which controls the food-producing areas. And being able to destroy them (or prevent trade) then becomes power.
Your society isn't going to be stable, and being away from the source of power (food) is going to be enslavement.
Even accounting for that, you'll have a complex web of transport; each node in the web has power over the next ones. They don't have to field an offensive army to lay siege to things further along the trade network, just disrupt trade going past them.
The instability of the situation will prevent stable societies from forming that depend on the food transport far away from the sources.
So, let us say we solve all of this. Society is out of a Malthusian Trap, it has a world-spanning Sail based (low rocket equation) transportation network, and it is web-like and/or has a form of law that is sufficiently strong enough that the food continues to flow, and everything is stable enough.
Then you won't stay Late Middle Ages very long. You'll have large populations not spending their entire lives scraping for foods; a society of artisans and crafters. You'll have an economic explosion of activity as innovation piles on innovation.
You are going to experience an Industrial Revolution or a Renaissance.
The Renaissance was the Merchant class displacing the land-owning Gentry and having a run-away wealth experience in Northern Italy.
The Industrial Revolution started in Netherlands, where the Malthusian Trap was defeated by a percent or so per year for a century, resulting in the wealthiest population in Europe (and possibly the World). Then the UK mimiced it, outpaced it, and then started a "virtuous" feedback loop of coal mining and steam power.
Basically, areas of Europe in the Middle Ages where most of the population wasn't dedicated to growing Food where areas that didn't stay very "Middle Ages"-like for very long.