The Mage thows a fireball, the fireball is so large and powerful it hits the castle walls. The castle walls are fairly large and heavy built from large boulders, it makes a crack in the wall and pushes a stone out. In this case the fireball accelerates to full speed after the spell cast and decelerates to a near halt when it hits the wall.
From what little physics i know the force that hits the wall equals the weight times the acceleration. The acceleration is slow enough for a common man over a 50 yard distance to avoid it. Meaning the weight of the ball has to be fairly high to break down a common castle wall.
So if the fireball is fairly heavy and according to newtons law of motion "To every action there is always opposed an equal reaction", the mage has to throw him self several meters back to propel the fireball towards the wall. But this is the fun part, he just stand as if he had just thrown a tennis ball.
How could this be explained in otherways except "because magic"?