I'll imagine two mechanisms for doing magic-
1). The mage draws energy from the field into them self and the projects it back out to create the magical effect
2) The mage draws incoherent energy from the field toward them self and then focuses and directs the energy into a coherent beam or patten to create the magical effect.
I'll address 2 first. In this mechanism, it's appropriate to imagine the mage as creating a " low pressure" zone near them self and then creating complex and subtle distortions that act as a lens to focus, shape and redirect the flowing energy (initially flowing from all directions toward the low pressure zone). In this imagining, there are two ways for magic armor to interfere with magic: first, such a full body layer of positive magic all around the mage could counter act the mages ability to conjure the negative flow needed to start the spell. Second, the refocusing of moving magic to create an effect requires an astoundingly subtle and complex lens. This lens manipulation ability is what really separates great experienced mages from novices. The presence of any magic in the matrix of the lens is going to change the dynamic of lensing the flowing magic. For complex and subtle magic, a mage would want a casting zone as free as possible from magical noise that might create defects in his lens. We can see how being wrapped in heavily enchanted ridged magical armor could badly distort the magic lens and prevent the mage from focusing the energy as needed to create the magical effect.
You can imagine all sorts of other magical device limits / aides resulting from this mechanism. Not spelling them out cuz I'm texting this and my thumbs wearing out.
Mechanism 1 seems to me to result in more of a partial block to magic than absolute. In this case we imagine the mage must open a physical channel to the field and stream sufficient energy into their own body. Great mages only include those who can draw and hold huge amounts of energy without killing themselves. As above, head to toe armor could simply block the mages ability to open the channel. However, just leaving ones arms or other surface unarmored, and hence unblocked, would seem to remedy this. At least, one could draw in energy in that case, just more slowly due to the reduced area of the collector.
The slow draw rate could be a problem for spells so powerful that the mage actually can't draw the full energy required for the spell into them self prior to beginning outputting it to create the spell effect. Imagine a lightning bolt. The spell caster draws in magic rapidly. Once he has enough to overcome air resistance and create an ionized channel between him and his target, he releases the energy to do so, then he continues to rapidly draw in and dump out energy in the form of electricity into the ion channel, causing damage to the target and preventing collapse of the ion channel. The mage can't draw the whole energy supply for this spell in before beginning because it would burn him to a crisp. However, if his ability to draw energy quickly from the field is retarded by armor, he could open the ion channel, but then see it collapse because he can't feed electricity into it quickly enough to hold it open.
So, with this imagining, there are lots of spells that are impossible while wearing armor - particularly those needing big energy levels and or prolonged streams of energy flow. However, there are also many magical effects one would imagine being possible in armor - particularly if the caster is wearing modified armor that leaves collection and casting areas on his body unarmored and unblocked. Yes, the casting time for these spells might be significantly longer, but they would still ultimately be possible in armor.
Hopefully, you can find inspiration in these two scenarios that give you justification for the limits you want to set and also allow you to weave some deeper consequences and effects that make the world your creating richer and more layered.