Let's do some actual math to try and estimate things. First, some basic genetics. Each founding member of your population brings with them 2 copies of each of their chromosomes. There are many thousands of genes on each of these chromosomes, and the vast majority of them all work perfectly in each of us. But, due to random mutations, some people have copies of genes that don't work. Often times this is fine, because you have only one broken copy, and your other copy works and is able to compensate for the broken allele. Geneticists call this haplosufficiency. What this means is that the broken gene, or bad allele, is recessive, while the working gene, or good allele, is dominant. The bad allele only causes an issue in individuals that get two broken copies. If you have one broken copy and one working copy you are heterozygous at that locus, and you are a carrier for the disease. If you have two broken copies you are homozygous for the disease and will be affected by it.
Most bad alleles are rare, because they are selected against by natural selection. A carrier for a disease gene is only at risk of having a child with the disease if they happen to mate with another carrier of the same disease. This is why inbreeding is bad. When two individuals that are closely related mate, they have a high probability of both being carriers for the same genetic disorders, and therefore of having a child with two copies of the bad allele, and therefore the disease.
So, math time. I'm going to simplify things a bit for the reader's sake as well as my own, but the results should still be reasonably close to the reality.
Let's say we begin with a population of size N. That means there will be 2*N total copies of each gene or allele in our gene pool. So if anyone in our starting population of size N is a carrier for a genetic disorder, that genetic disorder will exist within our population at a frequency of 1/(2N). The frequency of the good allele in the population will be 1 - 1/(2N). Let's call these frequencies q and p respectively. Now, there are 3 possible genotypes, or genetic combinations possible. 2 good alleles, 1 good allele and 1 bad allele, and 2 bad alleles. For any randomly shuffled population the probabilities for each of these genotypes are as follows: 2 good alleles = p^2, 1 good and 1 bad allele = 2pq, and 2 bad alleles = q^2. The reasoning behind these numbers should be fairly straightforward. The probability of having 2 bad alleles is equal to frequency of the bad allele squared. Using some simple substitution we now find that the frequency of a genetic disorder which was brought into the population will be (1/(2N))^2.
Let's try our formula out with an actual example. Let's say we have a starting population of 10. One of our 10 people happens to carry a mutation in the CFTR gene, meaning they are a carrier of Cystic Fibrosis. This means that 1/20 of all of the CFTR genes in our gene pool are broken. The chances of a child in the population receiving 2 copies of the broken CFTR gene and thereby having Cystic Fibrosis is 1/20 * 1/20 or 1/400 or 0.25%. Now, this doesn't sound all that bad right? The problem is that your starting population would be very lucky if it only had 1 carrier for 1 genetic disorder in it. A very recent paper estimated that the average person is a carrier for 1-2 recessive lethal mutations: http://www.genetics.org/content/199/4/1243.full. If each person in our starting population was a carrier for a single different recessive lethal genetic disorder, then each of those 10 diseases would kill ~0.25% of our future population (slightly less because sometimes they would co-occur).
Let's make things worse and say we only had a starting population of 2. If each of those individuals were a carrier for a single recessive lethal mutation then those bad alleles would exist in the population at a frequency of 25% and children would get 2 bad copies 6.25% of the time. With two diseases that means roughly one eight of the children would die from genetic defects.
Let's make things better and say we had a starting population of 100, each of whom bring in 1 recessive lethal allele. Each of these 100 diseases would now only occur 0.0025% of the time for a total of 0.25% child death.
However, this is only taking into account lethal mutations. There are likely many more mutations that could cause infertility, intellectual disability, and numerous other problems. I can't find any numbers on how many of these types of mutations the average person is a carrier for, but it's likely higher than the number for recessive lethal mutations as the selection against them would not be as strong.
A few extra notes. First, these inbreeding effects will gradually decline over time. Each time a child is born with 2 bad copies of a gene and dies, those 2 bad copies are removed from the gene pool. The worse the frequency of the genetic disorders are, the faster the frequencies of the bad alleles will decrease in the population. Second, the starting population size will also determine how many generations it takes before the population is sufficiently mixed that inbreeding even begins to occur. In a starting population of 2, the first generation will need to inbreed, but in the population size of 100, many generations would go by before anyone needed to procreate with someone at all related to them. Third, when the starting population size is small the outcome will also be highly variable. The numbers I calculated above represent the average outcome assuming the population gets neither lucky nor unlucky in which alleles get passed on to the next generation, but with a small starting population size a few unlucky inheritances of bad alleles could have disastrous complications later on, whereas some lucky inheritances of good alleles could remove all the bad alleles from the population early on. Small populations would also have a high degree of chance in how bad the inbreeding becomes.
While I didn't really provide you with a concrete number, I hope the mathematics will allow you to calculate your own starting population size given your definition of "relatively clean".