100-150 individuals with equal sex ratios is the number I have seen thrown around for human survival while avoiding serious inbreeding issues. This number could be lower for more diverse populations and higher for more homogeneous populations.
This group could keep track of their family histories strictly to try and maintain genetic diversity for as long as possible. Initially, this could start specifically by mating with those as un-like you as possible. Letting in a few outsiders every few generations would be helpful if they were genetically fit, or if they were genetically very different from the population inside. Or, if they had some kind of genetic resistance to a deleterious genetic mutation occurring inside the original group. Your group could also maintain records of genetic abnormalities to prevent two recessive carriers of a mutation from breeding. Yay punnette squares!
Having kids with your cousin (even your 1st cousin), genetically speaking, isn't actually such a big deal - the aversion to that is mostly cultural. Problems arise more so when families repeatedly marry their first or second cousins, or their brothers and sisters (think ancient Egyptian royalty or the last Russian Czars). So in some ways, it depends on your time scale, and the number of people you start off with.