Well, first thing first, you need something that is heritable and unique to females. Assuming that your magic-capable species isn't humans-as-we-know-them, this is easy: instead of using the XY sex-determination system, as typically found in mammals, you could use the ZW sex-determination system, as seen in many species of bird, instead.
In the ZW system, males are ZZ (the homogametic sex), and females are ZW (the heterogametic sex). This is the opposite of that seen in humans, where males are XY, and females are XX.
Given a ZW species, whatever genes control magic goes onto the W chromosome, and are thus carried exclusively by females. It is a gonosomal trait.
You could also, if you want to, do it the other way around: Stay with XY, allowing this to be about humans, but put something on the male chromosome which inhibits the phenotypic expression of the "magic gene" carried on the shared chromosome. That seems riskier, though, as at least some males would have the inhibition gene mutated to the point that they would to some degree exhibit the "magic gene" in their phenotype.
Few genes are simple on/off toggle switches. Many work in combination with other genes, often to determine a point on a spectrum rather than the existence or non-existence of a trait. At the very least, you want it to be a polygenetic, rather than a Mendelian, or monogenetic, trait, to allow for a distribution of phenotypic expressions.
At that point, you should have all of the pieces necessary: the "magic gene" is carried exclusively by females (thus cannot possibly be exhibited by males), and the amount by which it is expressed in the offspring (its phenotypic expression) can vary on a continuum. The rest is standard genetics.
You could also look at epigenetic inheritance, in which the genetic code is identical but the phenotypic expression differs. And of course, nothing prevents environmental factors from playing a role; it's possible that all females has the ability, to varying degrees, but it needs to be cultivated from an early age to become useful.