On Colonization
To start out, you're not really going to get a population of 5000 out of 50 in a reasonable timeframe (less than several generations). Also, 50 people is way too few; it doesn't give you enough workforce to start producing your own food within the same season, much less build a barony.
In my country, we had something we call "internal colonization" in the mediaeval times. The land was mostly dense forests, and when the local lord wanted to develop some of his real estate, he would recruit a bunch of people to start a new settlement somewhere in the wilderness.
The understanding was that the colonists would found a new settlement that would owe feudal taxes to the sponsor. They would, as per the feudal contract, have a hereditary lease on the land, but to get the thing started, they would get a tax break for a period of time (between 8 and 20 years). This period was called a "reprieve", and the villages would consequentally have the word "lhota" in their name to signify that.
Of course, this would start a village rather than a barony; you would at most get a knight ruling there and not much in the way of troops. But it indicates that if you assemble enough willing people with the right equipment (say, 15 families), you'll have a self-sustaining settlement in about 10 years. If we want to have a barony with a castle, troops, and 5000 inhabitants we need to go a bit further.
So you want to start a barony
Let's take a look at your requirements:
- population of 5000
- a castle
- 1000 troops
As they are, the numbers seem to line up pretty well once you reach a stable state. If you're willing accept irregulars as troops (who, with the proper kind of background, can be utilized to great effect), you can hit your levy target if you have that population. 5000 people is also almost a city (as such things were reckoned in the middle ages), so you shouldn't have trouble securing the maintenance of the castle.
The trouble is, castles are frigging expensive and take decades to build. In order to get one, you would need serious motivation to get started on it right away, and a revenue stream to support the construction. If there is such a revenue stream, however, it also doubles as a motivation, and with magical tools, you may be able to cut down on the construction time.
Since we are talking middle-ages level, agricultural efficiency dictates that to keep your barony from starving, the majority of people should be employed in agriculture (see this nice essay on medieval demography for details). What you would eventually get would be a (castle) town as the economical centre and a bunch of villages scattered around, growing the food.
There are three ways that I can think of to get such a barony from expeditions
The long game
Say you're a count living next to the wilderness who wants to expand his domain. You don't have much in the way of money, but you're savvy and willing to play the long game. That's good, because doing it this way will only cost you time.
So you gather a couple of settlers to go into the wilderness and start a village. It will be a bunch of farmes, some woodsmen, maybe a blacksmith; they will go out into the forests, do a bit of slash-and-burn and start an agricultural village. Since it can be dangerous out there, it might be a walled village with a pallisade surrounding it, but a village none the less.
Now, rinse and repeat. Each of these expeditions cost you some money (you get the settlers for free just by offering land; if you only send 50 at a time, you likely won't run out of volunteers), so however much you are willing to spend at once probably sets the limit on "parallelisation".
After 5-10 years, depending on the conditions and what magic you have and such, each of these settlements becomes self-sustaining enough to pay taxes. You're going to need, to eyeball it, about 75 to reach a critical mass, when you have enough people to economically support a city. Now your master plan comes to fruition.
You pick a person you like and authorize them to found a city. You give them town privileges and get a bunch of journeymen specialists to come by promising them head positions in the new guilds you're going to found. You build a town, economically supported by the surrounding countryside, and you've mostly reached your population goal by now, with the only thing left being the fort.
The catch here is that it will cost you. There is probably motivation to build walls, since this is after all wilderness, but it will eat up a lot of the town revenue, and you might end up with a poor-ish city holding rather than a CK2 barony.
If you have a valuable cash crop, you can send in a couple more village expeditions, but these will also require sustenance and drive your population up, and also take time to become self-sufficient.
All told, you can put up a walled city with maybe a citadel for the local lord in 25 years or thereabouts of concerted effort, assuming nothing goes catastrophically wrong. You'll probably end up with a somewhat higher population in order to support the city, but on the other hand, cities will provide more levies (mostly in the way of pikemen and such), because they tend to be more independent-minded and jealous of their privileges. And if you set things up right and the conditions are favourable, it will continue to grow on its own.
The expensive way
On your teritory, there is an important crossroads of trade routes that you want to keep safe (and extract tolls from the traders for the privilege of such security). If you have the funds and magical equipment, you contract a bunch of people to build a full-on castle nearby. You hand it off to someone you like and give them a garrison to go with it.
Lots of castles in Europe were founded this way; the tolls provide a revenue stream (or you can resort to banditry if you're so inclined) which can pay for the upkeep and eventually, your castle will develop a castle town to help with the maintenance and profit off the trade. You can help this process by setting up a few villages in the surrounding countryside, supplying the castle with cheaper food and helping its growth.
This is very intensive in terms of the original investment, but you start out with the castle first and only need to accumulate the population later. The growth here is again going to be driven primarily by immigration, the rate of which will depend on the profitability of the trade route and can be boosted by starting villages.
The castle will have a sizeable garrison, which will provide the core of your army here; the rest you need to recruit from the town or villages. With the equivalent of "power tools", I'm going to guess it will take about 5 years to get the castle fully up and running. The craftsmen who participated in the construction will probably hang around for maintenance and form the core of the castle town; the rest of the population will trickle in depending on how good the business is; this may take anywhere between 10 and 100 years depending on how profitable the trade route is and how much you're helping by systematically founding new villages.
You can also do it this way if there is no trade route and you just want to fortify a strategic location; but without the revenue stream, this will eat up insane amounts of money until you accumulate enough population by systematic settlement as per the previous option.
If you're lucky
Your prospectors have uncovered a deposit of unobtanium in the wilderness in your teritory.
You lease the mineral rights to someone you like and give them a sizeable loan to start a mining town. If the thing you're mining is really profitable, you'll get a bit of a gold rush, so there will be no problems getting volunteers.
An expedition of a couple of hundred miners, craftsmen adventurers and assorted personel venture into the wilderness and set up camp around the deposit. About half of them throw up makeshift fortification, in case someone decides to forget their manners and take the deposity by force; the rest get directly to mining in order to pay for the construction.
Eventually, you have a mining town running. This will be fairly rich, and if you did the smart thing and granted it to a trustworthy, savvy nobleman, he will immediatelly put to building a castle to protect his newfound riches. With the power tools and enough money, he will in a couple of years have a castle with a sizeable courtyard, to house the inhabitants of the surrounding countryside in times of crisis, and probably city walls too.
The wealth of the town will attract more immigrants, and the ruler himself might start founding villages to supply the mining town with cheaper food and drive the population up.
Again, the population will be the limiting factor on how much troops you can field, but with money, you can easily hire professional soldiers until your population grows sufficiently; even then, you might want to keep a stable retinue, because mining towns make tempting targets.
All told, with power tools you can have the mine, city and the castle running in 5 years tops, with the population growing steadily in the next 5-15 until you reach your goal. The mines will run dry eventually, but by then you should've built up enough economical momentum to keep the whole thing going.
Bottom line
Depending on what you want and what your means are, you can probably set up a barony with a series of settlement expeditions in 10 years under the right conditions, or up to 100 years otherwise. The driving force behind the growth is always going to be immigration, though, and the limiting factor will be either food to support the population, or the money to purchase it from elsewhere. Having a revenue stream in the form of cash crops, trade routes or mines helps immensely.