You could have it that magic works from a "second level" physical factor. By second level, I mean something like the way you have the really obvious physical factor "speed" - easy to see in place everywhere about you, and to remember your own experiences - and it would be a first level factor, and then you have a second factor, well-known and experienced by many, etc., called "acceleration." Without speed, there is no factor called acceleration, so speed is the first level factor, then acceleration is a change in speed and so a second level factor.
Building on that, consider two magnets. Put North vs. South and they stick together through their magnetic force. And never do anything else. Move them separately so that their forces can interact and you generate electrical force if you have a suitable "vessel" for it (a circuit in this case). Whumped together so they are locked, there is no changing interaction and no useful electrical force. Kept separated and changing place relative to each other and it is possible for their interaction to generate a seemingly wholly different force that has interesting capabilities of its own.
So... consider the idea that your magical power factor depends on a second level factor that requires first level factors moving relative to each other. In that case, you could observe three basic outcomes:
1) Things very close, like the whumped together magnets, haven't enough relative motion to generate enough of the second level factor (magic). So the magic user would, technically, get some power from his own planet, but soooo little of it, no one ever noticed or measured that teensy bit.
2) Farther away things would obviously move relatively to their planet. Look at what even the moon does with tides. Since YOU get to decide the background, it could be OK that the magical power is significant at solar system distances. So everyone would notice the magical power, and they would realize - lol, wrongly, but hey, it's magic, not science, right... or maybe it's the scientific revolution period for magic on your world and they WILL come to realize it's not the planets and so on themselves, rising and falling in contributed strength based soley on distance, or weird properties that don't seem to exist on the populated planet, but rather the change, the relative movement - but anyway, they realize the magical power they use "comes from" the other solar bodies.
3) Distance COULD be a factor even on a near solar scale. So for us, Pluto would be noticeable nowadays, but when it runs out another billion miles in its orbit, it might not be very noticeable even when on the same side of the sun. And it could be why extra-solar things have little impact.
Naturally, there would be other things to consider in. Maybe the sun's composition is too low on the periodic table to have much effect so it doesn't swamp everything out (because anywhere habitable in the solar system might be kind of like the "magic user-his own planet-not enough change to give much magical power" thing. It could turn out some things could mask (insulate) the power or make the power futzy, like Jupiter having tons of moons and them making its contribution rise and fall rapidly and in weird ways. Just studying that could be a career. Magical items could be of interest, not really how they hold/contain the power, but more the idea that all people, even animals, actually have magic-use capability but very little. VERY little. So they don't even power up from what powers up an average magic-user. But if a magic-user concentrates the power into an object, like ramping up energy in fast discharge capacitors is used for powering lasers to stunning power levels, the average person, even animals or even plants, could then be handling a power level that even they can make work. So a +1 sword works well for me, or might be a +6 sword for someone good with magic even without the item's stored power. They would also run down...
Hence also propitious moments for some events and uses of magic. Just the right combination of planets and other astrological faves. And given they think the power is actually directly coming to them, not being generated all around them and they just use it well, worshipping various sources as gods could be a natural outcome as well.
Perhaps something like that. Simple without being awkwardly simplistic. Simple to grasp the idea when explained, haltingly in the text if a book, or game if a game, let's one happily forgive the reality of magic being so much hoo-ey, and let's one plunge right in and enjoy. But the simpleness hides your ability to slowly reveal a deep texture and outcome profile, a truly rich background to work with as author or programmer. And endless supply of operational situations. After all, look at Star Trek. We accept the idea it all works because it all seems so plausible. No one in the shows has to step to a chalkboard and teach the fans that it all starts with being able to generate and control stunning levels of energy and that transporting is actually creating a warp field around the person and moving that person like a starship does in its warp field, and stopping them just right and dropping the field... who cares? It is presented so nicely you slide right in and it's real and lends to the milieu's amazing richness and fun. Same here. Well, if I do say so myself that is.