Pseudo-science and real science to follow
First, a caveat - I don't know much about genetics, but I do know that it is complicated (read artistic licensing can be applied more or less liberally) and I'm really good with the Punnet Squares. Depending on how much you want to be tied to real principles of genetics, there are LOTS of ways to do this. The variables of note are (1) the number of "genes" used, (2) the number of "traits" used, and (3) how the geneotypes manifest phenotypes (including traditional dominant vs. recessive and more hypothetical (read likely not realistic) dominance schemes), and one assumption: possessing magic is beneficial and culturally desirable (this is import for consequences), and a few very, very needful simplifications. I shall refer to a person who has inherited magic as a Mage, a person who has not inherited magic as a Non, and the five magical branches as A, B, C, D, and E respectively.
1 gene, 5 traits, magic is recessive
$$
\begin{array}{c|c c}
& \text{A} & \text{B} & \text{C} & \text{D} & \text{E} \\ \hline
\text{A} & \textbf{AA} & \text{AB} & \text{AC} & \text{AD} & \text{AE} \\
\text{B} & \text{BA} & \textbf{BB} & \text{BC} & \text{BD} & \text{BE} \\
\text{C} & \text{CA} & \text{CB} & \textbf{CC} & \text{CD} & \text{CE} \\
\text{D} & \text{DA} & \text{DB} & \text{DC} & \textbf{DD} & \text{DE} \\
\text{E} & \text{EA} & \text{EB} & \text{EC} & \text{ED} & \textbf{EE}
\end{array}
$$
Inheritence
Mage/Mage
$$
\begin{array}{l|l l}
1 & \text{A} & \text{A} \\ \hline
\text{A} & \text{AA} & \text{AA} \\
\text{A} & \text{AA} & \text{AA} \\
\end{array} \space{}\space{}\space{}\space{} \begin{array} {c| c c}
2 & \text{B} & \text{B} \\ \hline
\text{A} & \text{AB} & \text{AB} \\
\text{A} & \text{AB} & \text{AB}
\end{array}
$$
[1] A pair of Mages of the same branch will always produce a Mage of that branch
[2] A pair of Mages of different branches will always produce a Non
Mage/Non
$$
\begin{array}{c|c c}
3 & \text{B} & \text{C} \\ \hline
\text{A} & \text{AB} & \text{AC} \\
\text{A} & \text{AB} & \text{AC} \\
\end{array} \space{}\space{}\space{}\space{} \begin{array} {c| c c}
4 & \text{A} & \text{C} \\ \hline
\text{A} & \text{AA} & \text{AC} \\
\text{A} & \text{AA} & \text{AC}
\end{array}
$$
[3] A Mage and a Non with no common traits will always produce a Non
[4] A Mage and a Non with a common trait will produce a Mage of that branch (0.5) or a Non (0.5)
Non/Non
$$
\begin{array}{c|c c}
5 & \text{C} & \text{D} \\ \hline
\text{A} & \text{AC} & \text{AC} \\
\text{B} & \text{BC} & \text{BD} \\
\end{array} \space{}\space{}\space{}\space{} \begin{array} {c| c c}
6 & \text{A} & \text{C} \\ \hline
\text{A} & \text{AA} & \text{AC} \\
\text{B} & \text{BA} & \text{BC}
\end{array} \space{}\space{}\space{}\space{} \begin{array} {c| c c}
7 & \text{A} & \text{B} \\ \hline
\text{A} & \text{AA} & \text{AB} \\
\text{B} & \text{BA} & \text{BB}
\end{array}
$$
[5] Two Nons with no common traits will always produce a Non
[6] Two Nons with one common trait will produce a Mage of that branch (0.25) or a Non (0.75)
[7] Two Nons with both traits common will produce a Mage of one branch (0.25) or the other (0.25) or a Non (0.5)
Consequences
This would likely produce a natural caste system where 'noble' houses intermarry with noble houses of the same magical branch and carefully control the bloodlines. The common folk will periodically produce a mage by accident, quite scandalous! Some of these would periodically get married into the noble lines because the heart wants what the heart wants or to deliberately help with genetic diversity.
These nobles would likely blend (a mixed region/country), resulting in few common born mages (because of genetic diversity), or separate (mono-branch region/country), leading to a higher frequency of common born mages (due to lack of diversity of nobles - sewn oats and all that). Mono-branch regions would have an incentive take prisoners from other mono-branch regions to diversify (and oppress) the common stock. Mixed regions seems the most likely if different branches can accomplish different things, but if similar effects can be achieved with each branch then both would be likely to occur.
1 gene, 6 traits, magic is dominant
As above, but a 6th trait N is introduced. All co-dominant pairings can be exclusively Nons or exclusively Mages if following some dominance rules (AB always manifests as A, etc. -- several rock, paper, scissors, lizard, Spock style diagrams could map out all pairings) or some combination (AB always manifests as A, AC is always a Non, etc.). All A-E/N pairings are a Mage of the appropriate branch. All NN pairings are Nons.
Consequences
If co-dominant pairings are Nons, Mages are still rare, similar as magic recessive above. Pure Mages (AA) could be more powerful (due to purity) or less powerful (due to latent effects of N, perhaps a larger pool of power from which to draw -- hence NN are Nons that could theoretically be considered Mages with large pools to draw from but no way to expend it). If Pure Mages are more powerful then there will be little dilution in the bloodlines. If *N Mages are more powerful, bloodlines will be mixed and some children will be Nons and thus possibly shunned, cast off, executed, exiled, used for purely political marriages, etc.
If co-dominant pairings (AB) are Mages, then Mages will likely be VERY common. If Pure Mages (AA or AB) are more powerful than Non-Mage mixes (*N) then Nons will be second class citizens in many places. If Pure Mages are weaker than Non-Mage mixes then Nons will valuable property/breeding stock in some places and key figures of important houses in others.
2+ genes, 3+ traits
Mages have one or more genes that determine if they are a Mage and one gene with 3 or more traits that determine what Branch of magic they have. If magic is recessive, the more genes required to be a Mage the fewer Mages there will be. If magic is dominant, the more genes that create Mages the more Mages there will be. With 3 traits, you would have 6 distinct pairs (AA, AB, AC, BB, BC, CC) or 8 permutations (treating AB and BA as different, etc.), which would be enough to map out to 5. The more combinations belong to a branch the more populous that branch will be, so this set up would be ideal if you wanted a magical ruling class (rare branches) and a magical laboring/middle class (common branches). You need 5 (or a multiple of 5, but 5 would be the most simple) to have all branches be roughly equally represented (see rock/paper/scissors/lizard/Spock comment above for how to chart that).
A pairing of mixed people from mixed branches
$$
\begin{array}{c|c c}
1 & \text{MA} & \text{MB} & \text{NA} & \text{NB} \\ \hline
\text{MA} & \text{MMAA} & \text{MMAB} & \text{MNAA} & \text{MNAB} \\
\text{MC} & \text{MMCA} & \text{MMCB} & \text{MNCA} & \text{MNCB} \\
\text{NA} & \text{NMAA} & \text{NMAB} & \text{NNAA} & \text{NNAB} \\
\text{NC} & \text{NMCA} & \text{NMCB} & \text{NNCA} & \text{NNCB} \\
\end{array}
$$
M is the gene for Mages and N is the gene for Nons. If magic is dominant, MN could be a Mage or a Non depending on how you treat mixed branches (if A or B is dominant over the other, then it is still a Mage, if traits are co-recessive then it is a Non). If magic is recessive, MN is a Non regardless of branch traits. For multiple magic genes, it is similar, but the portion that is magic/non-magic shifts appropriately.