WALLS. Big beautiful walls.
With gates, written permits with wax seals.
What are they for? Restriction and control of certain areas. Restriction and monitoring of movements.
The Keep itself, surrounded by the highest, strongest walls will be a heavily restricted area. Only the officially recognized guards, officials will be allowed to enter. Otherwise, only those with written permits by the Lord himself. Customs may be employed that guests can't bring weapons, so on.
Town around the keep. It will have smaller walls, not so much for military defense but for better control of normal people and goods. It allows taxation, search for wanted criminals, ... A census can be kept of those with permanent residence and even visitors. A much simpler written permit allows entrance, written by numerous officials tasked to oversee it. Residence permits or privileged can avoid the visitor lanes and enter without undergoing search... or much lighter one.
Walls of the valley. A simple wood and mud wall, with patrolling guards are enough to curb movements. Those who must avoid official routes will find ways around it. For simple people, armed guards and fear of death are good enough deterrent.
Mini Frame Challenge:
Access points, the Gates of Valley, won't have traffic in the thousands every single day. Even if they have, they can wait to be processed. Few people were in that much of a rush, so they can't wait their turn... and those who truly couldn't, obtained special permits in advance (perishable traders, government officials, ...).
What they were most concerned in those times, were the movement of masses (they can be a threat), movement of goods or contraband (tax avoided: their money!!!) and a few high interest personage for various reasons (but simple portraits and descriptions of their likeness will be distributed in advance).
In case of above average number of travelers, additional guards and officials can be redirected to help with inspections. Otherwise, those without authority or power can wait their turn patiently. Their comfort and needs were rarely a priority or consideration.
The farming migrants will disperse into small villages around the valley. They have no reason to be crowded in one place (after their leader offered an oath of loyalty and servitude to the Lord and through him the Country and current Ruler). It is a logistic, hygene and many other nightmares at once. Chinese had big cities, western countries not so much until industrialization kicked in.
The merceneries will semi-integrated into the standing army, housed in barracks for better control and their leaders' family "gifted" with permament residence inside the Keep (maybe employed as special status semi servants) or in the Town, with much more restricted movement permits.