I think you have two options: 1) import all your food; 2) grow your food.
Ancient Rome or medieval cities can be a good example here. At its peak, the population of Rome was about 1 million people. Obviously, they could not grow their food themselves. So, they were importing almost all of it.
In case of Rome, they had the entire Empire to provide them with grain. You can follow their example and establish a protectorate over the nearby lands and collect tribute in foodstuffs from the surface population. You can design it as a "monster protection fee". To spice things up you can ally trolls and alike with more human-like species in a quest of establishing control over farmers.
Another approach is based on trade (similar to medieval towns and even modern cities). Your underground towns can become centres of free trade where anything can be bought and sold. Your business model should be based on the idea of providing unique goods and services. For example, it can be a place to hire mercenaries, find a rogue mage, or buy unique armour, weapons, and jewellery (elves+dwarves would make a fantastic design team). I would also look into spider silk. If your caves are the only place where it can be produced and made into cloth or armour, you have a monopoly.
With income generated from the trade, your underground city should be able to buy supplies.
2. Food production
This can be tricky. A lot depends on available magic and technology. You also need to think about balancing the diet. If biology of your underground dwellers is similar to human you must provide them with sources of dietary fibre, carbohydrates, protein, fats, and micronutrients (vitamins and minerals). Unless it is a dystopian realm where diet related sickness is a feature.
It might be possible to create something like vertical farms. If your underground citizens can make mirrors, they can create an elaborate system of mirrors reflecting sunlight into caves. This would allow growing food on shelves. Hydroponics (water mist in particular) is the best growing method in this setup, but I am not sure if you can do it with pre-industrial technologies.
In addition to vertical farms for vegetables and grain, you should build mushroom farms for a healthier diet. Insects and worms should also be included to provide protein. Small animals like hares can be raised in cages. They are low maintenance and a good source of meat. However, the mean is extremely lean, so you must provide your citizens with some oil or fat to avoid rabbit starvation. If you have access to the surface in a mountainous region, goats can also be an option. Goats are great for meat, milk, wool, and hides.
As @ZioByte suggests, algae pools can be another source of nutrients, especially protein. They still need light and quite bright for rapid growth. However, algae can feed not just fish but also humans, poultry, and livestock. An additional benefit is that some of them can use diluted human urine as a growth substrate. Hence, you can keep your city clean.
Fish ponds stocked with species similar to tilapia are also a very attractive option. Tilapias grow fast, easily tolerate overcrowdedness, and are very sturdy. While real tilapias are sensitive to water temperature, it might not be the case in your situation.
Pigs would be another great addition to your city. They are omnivorous and can consume almost all food waste produced in the realm. Their excrements can be in turn used to feed algae and shrimp (the latter is done in China).
Throw in some hunting and gathering and your population should be fully covered in terms of dietary needs.
One more thing to consider when it comes to growing food is how centralised the production is. You can have underground farmers or huge government operated factories. Your city might sponsor farming infrastructure (mirrors and irrigation) and then tax farmers (or all citizens) or farmers might be left to their own devices.
Magic makes all of this much easier, especially if your mages can create artificial lights, irrigation systems, and so on. A huge drawback is that they become de facto rulers of your dark realm. Everybody would depend on them for food. Unless they are needed only to set the production up.
A combination of the two approaches would give you the best chances of feeding your population.