This is the enemy infantry line.
This is our archmage casting fireball.
Area of effect spells can be found in most magic systems. However, whatever the exact effects of the spells are, hitting a tightly grouped crowd is always going to do most damage. In premodern battles strong formations, which don't break are what everyone wants. Those formations usually consist of tightly grouped infantry, as shown on the first picture.
While in our world line-infantry combat was ended by the advent of the machine-gun, I suspect that in a world with magic, AoE spells will have the same effect much earlier. The question is how warfare would evolve to deal with these spells. Some solutions I came up with are:
Power Level Control
Just keep the mages and the magic system free of AoE spells. However this isn't really what I want and probably won't even work. Getting creative with some alchemical neurotoxin and telekinesis will be as bad as an all out arcane firestorm. Additionally I want my mages to be powerful.
As a side note, while a large percentage of the population in my setting can learn magic, few actually do. Of these, most are weak witches, healers, charlatans and shamans. Few have the potential and fewer get the education to become proper wizards. Fewer of these actually have the inclination and talent to become proper war mages. However, most nations do have at least some of these around.
Using countermagic to neutralise magic will only work if both sides are equal on the magical front. Otherwise smashing the enemy with spells is preferable. Of course, if antimagic is an absolute counter to magic this will work, but will also make mages really weak.
Shielding the troops with protective gear and magic might work for rich elite troops, but not for the peasant infantry. Furthermore magical resistance can be circumvented. While a fireball might fizzle out when it hits an antimagic zone, a 500 kilogram boulder propelled by a telekinesis spell will carry on as a ballistic projectile.
Magical resistance might be somewhat useful, but will ultimately only force the mages to adopt new tactics.
Spread Out The Troops
The tightly packed front line is what make AoE spells so effective. If there are fewer soldiers per unit of area, AoE spells become less effective. However a loose melée front line fails its original purpose, which is allowing the warriors to mutually support each other. This would mean that melée infantry would disappear from large battles and that loose formations of missile infantry with secondary melée armaments would dominate.
Assuming that spells announce themselves before they hit, dodging could be a viable strategy. Cavalry comes to mind here. Heavy cavalry to harness the ranged infantry and mages and horse archers as an upgrade of the ranged infantry.
Since AoE spells could be avoided by taking cover and since traditional defences against cavalry are gone, entrenching one's position on a battlefield becomes increasingly important. Defensive fighting positions with a special focus on cavalry defence might become relevant on the battlefield. This would also mean that bows and crossbows, which can be operated kneeling would become relevant weapons. Caltrops and spikes, maybe doubling as javelins, could be used to enhance the defence of such positions.
While I think that line-infantry tactics would still be used in minor battles and skirmishes when no mages are around, major battles with mages involved might use tactics as described above. Did I miss something? Are there better ideas for adapting pre-modern armies to AoE magic?