Allow me to shed some light on this subject (turns on lightbulb).
Brief Overview of Magic
These powers manifest themselves any time from birth to the end of puberty. These magical powers fall under seven broad categories. Air, water, earth, plasma (fire/lightning), plants, light, and animals. Animal magic is tricky, because living things resist magic. But, many animal mages can influence animals, and some can control dead animals (this is called necromancy and is a big societal no-no.) The strength and scope of magic varies from individual to individual. But, these are the most common in order. Earth, water, air, plants, animals, plasma, and light.
These magical capabilities are like the show Avatar, the Last Airbender, in how controlling magic works. However, in my book magic is much less powerful. For instance, an earth Mage may only be able to influence as much dirt as they could lift, and lifting this dirt would take about as much energy as doing it by hand, except you don't have to touch it, and it could be a perfect brick shape. No force is exerted in separating the dirt or material from other similar material attached to it. So if you were to pull this rock in two, that would take little effort, other than the force of pushing the pieces apart.
Magic is like a muscle that is really hard to develop.
Magic on a larger scale
The level of magic described above is only about 75% of magic users. Everyone else is more powerful.The strongest people can do things like shoot lasers, uproot trees, be human flame throwers, make zombie hordes, create small tornadoes, cause rovers to overflow or to break dams, or cause small earthquakes. This is about 7% of the population. Everyone else is in between.
Magic on a smaller scale
I'll just make a list of some possible fighting applications of magic. Ice daggers, water whips, wind influenced arrows, starting fires, shooting small fireballs, throwing stones, moving stones, strangling people with vines, tripping up enemies with plants, scaring/exciting horses, and flashing people. (With lights!) These are of course by no means the only applications of magic.
Magic Resistance
Living things are resistant to magic, proportional to their intelligence/self awareness. So it would be easiest to make a swarm of spiders, then harder to make boars attack (your enemies, and not yourself), and nearly impossible to make someone slap themself. There is a type of stone resistant to magical tampering, which is found deep underground, near bedrock. Also, bones are immune to magic.
Note - These things are resistant to magic being influenced directly upon it, not magically influenced material. A rock thrown by magic at someone is just as effective as a rock thrown without magic at the same speed.
Now that You've heard me drone on and on, let's get back to the original question. With magical abilities such as these, how would the architecture and design of castles change to combat these threats? Would castles even be practical?