The magic in my setting arises from inbreeding. However to gain the powers a person must be in-bred. How long would it take before such problems arise from inbreeding if the people continued this practice despite the problems?
There is no hard number for telling after how many generations interbreeding will become problematic, it is a matter of statistic and thus probability.
For reasons which I think are obvious, the only available studies have been done on marriage between cousins.
Cousin marriages have genetic aspects arising an increased chance of sharing genes for recessive traits. The percentage of consanguinity between any two individuals decreases fourfold as the most recent common ancestor recedes one generation. First cousins have four times the consanguinity of second cousins, while first cousins once removed have half that of first cousins. Double first cousins have twice that of first cousins and are as related as half-siblings.
In April 2002, the Journal of Genetic Counseling released a report which estimated the average risk of birth defects in a child born of first cousins at 1.1–2.0 percentage points over an average base risk for non-cousin couples of 3%, or about the same as that of any woman over age 40. Put differently, a single first-cousin marriage entails a similar increased risk of birth defects and mortality as a woman faces when she gives birth at age 41 rather than at 30.
Repeated consanguineous marriages within a group are more problematic. After repeated generations of cousin marriage the actual genetic relationship between two people is closer than the most immediate relationship would suggest. In Pakistan, where there has been cousin marriage for generations and the current rate may exceed 50%, one study estimated infant mortality at 12.7 percent for married double first cousins, 7.9 percent for first cousins, 9.2 percent for first cousins once removed/double second cousins, 6.9 percent for second cousins, and 5.1 percent among nonconsanguineous progeny. Among double first cousin progeny, 41.2 percent of prereproductive deaths were associated with the expression of detrimental recessive genes, with equivalent values of 26.0, 14.9, and 8.1 percent for first cousins, first cousins once removed/double second cousins, and second cousins respectively.
This is a somewhat limited answer.
By and large every person has a certain number of genes. Of every gene there are two copies. Normally there is one recessive gene and one dominant gene. While mutated genes may create an illness, the redundant gene usually covers for it. If a person ends up with two of the same gene, they will have no redundancy, such that a defective gene would make them ill.
By and large, the closer the relationship, the more likely they share a gene. Thus their offspring is more likely to have paired genes, therefore more likely to have deformities and other problems. If we'll take the most extreme example of exclusively brother-sister pairings, it is likely that there will be defects already in the first generation. Conversely, many genetic defects make offspring unviable or infertile, so while the worst defects would breed out, many minor ones would remain, with later generations having less non viable offspring and more deformities and defects, the longer this goes on.
The exact amount of deformities will depend somewhat on the original ancestors' diversity, and on their genetics. If you had two people with no genetic defects at all as initial ancestors, theoretically nothing bad should happen for a couple generations. After that minor mutations in different individuals will begin to make trouble.
Of course the obvious way to avoid genetic problems arising from inbreeding is to combine modern knowledge of genetics with strong magic, to eliminate bad genes from the eggs and sperm of the members of a magical lineage. If their eggs and sperm are reduced to those which only have non harmful genes, any possible genetic combination will have no bad genes.
Thus there can be as much close inbreeding as desired, once harmfull genes are eliminated from the original gene pool.