Let’s start with the easiest of your questions:
Are any multi-planetary binary-star systems known?
There are, in fact, quite a few:
- 55 Cancri: Five planets orbit 55 Cancri A (a G-type yellow dwarf). Four are gas giants, three of which orbit inside 1 AU, while the fifth is a super-Earth orbiting at about 0.015 AU. The two stars, by comparison, are separated by over 1000 AU. Stability is quite clearly not an issue here. 55 Cancri B is a red dwarf/
- HD 20781/HD 20782: These two stars form a binary system of two G-type stars. The former has two Super-Earths orbiting it at less than 1 AU, while the latter is orbited by a gas giant at about 1.80 AU.
- Kepler 47: Kepler 47, a binary composed of a G-type star and a red dwarf, contains three exoplanets, all of which orbit Kepler 47A. They’re gas giants, orbiting within 1 AU.
- NN Serpentis: Composed of a red dwarf and a white dwarf (a stellar remnant, technically), NN Serpentis contains two gas giants, orbiting at 3.4 and 5.4 AU. The planets are circumbinary planets, as the two stars orbit close together.
As you can tell, there’s a mix of planets in P-type orbits (circumbinary) and S-type orbits (around just one star). The planets in S-type orbits seem to orbit close to the primary. It could be argued that this is an observational artifact; after all, it’s much easier to detect giant planets close to their parent star, using transit and radial velocity methods. I would guess that this plays a role in the lopsided data we have, just as it plays a role in the distribution of all the exoplanets we know of.1
However, there is another factor. Planets in S-type orbits do have a maximum semi-major axis, as first explored by Graziani & Black (1981). This distance is accepted to be about one-fifth of the separation between the stars. Likewise, for planets in P-type orbits, there is a zone of (according to the above page) one third to 3.5 times the separation in which stable orbits are not possible (see e.g. Donnison & Mikulskis (1991)), though Wikipedia cites the more recent Welsh et al. (2013) in extending the innermost stable orbits to 2-4 time the stellar separation.
These are some basic ranges; Hypnosifl also suggested Holman & Wiegert (1998), who studied the long-term stability of both S-type and P-type orbits, allowing for non-zero values for the eccentricity $e$ of the orbits of the stars. Thus, their empirical formulae (with coefficients derived using least-squares techniques) may be more accurate than models which assume that $e=0$. I’d highly recommend consulting them if you want to do some basic calculations for your stellar/planetary system.
An even better resource might be Jaime et al. (2014). They discuss the applicability of stability to habitable zones, including the continuum between habitable zones around each star to habitable zones around both stars:
Additionally, like Holman & Wiegert, they discuss the impact of stellar eccentricity on the boundaries of stable orbits.
The authors look at the cases of three specific stellar systems and try to define whether or not a planet can orbit stably inside the habitable zone(s). Take, for instance, the case of HIP 80346:
The red line denotes the outermost stable S-type orbit, and the grey areas denote the habitable zone. The stars are shown in black. Using their analysis, they were able to show that only the secondary star can host planets in stable S-type orbits in its habitable zone, because the primary star’s maximum stable orbit is inside the inner edge of the habitable zone.
Now that we seem to have looked at the limits of orbits, let’s take a look at what you’re actually likely to find in a binary star system. Quintana & Lissauer (2007) is a good start, in terms of terrestrial planet formation. Here’s a quick summary:
- They began with a large number of planetesimals already formed. 14 compose half of the mass of the disk (each with a mass of about one tenth that of the Earth), while 140, each with a mass one tenth of those planetesimals, compose the other half of the mass.
- In a simulation of the Alpha Centauri system2, with a protoplanetary disk surrounding Alpha Centauri A, five terrestrial planets with masses at least that of Mercury formed in orbits with semi-major axes less than two AU that remained stable for the duration of the system, 200 million years. However, with a tiny perturbation in the initial position of one of the planetesimals, only four planets form, all within 1.8 AU. Furthermore, if the disk is given an inclination of 15 degrees, only three planets form, because mass is lost. In the simulations, 3-5 planets formed in S-type orbits around Alpha Centauri A under a wide range of conditions; for Alpha Centauri B, the range was 2-4.3
- In the case of general S-type orbits in “wide” binary systems, two factors that influenced planetary formation more than anything else: the stellar mass ratio, $\mu$4, and the periastron distance (the closest separation) of the two stars, $q_B$.
- In the case of P-type circumbinary orbits, close binary stars yielded terrestrial planetary systems similar to the Solar System, especially if giant planets were added. However, in some wider binaries (stellar separations of about 0.5 AU), drastic mass lost led to only one planet. As would be expected, the most important factor here is $Q_B$, the apastron distance (the furthest separation), as it determined disk perturbations.
- The masses and orbital eccentricities of the planets are not affected much given that there are two stars in the system (for both P-type and S-type orbits), given suitably stable orbital ranges for the $q_B$/$Q_B$.
Here’s Figure 1, the results of some of the simulations for planets around Alpha Centauri:
Here’s Figure 2, the results of some of the simulations in the same conditions but with that tiny change in the orbit of one of the planetesimals:
I’d also like to mention Bromley & Kenyon (2015), who attacked the problem analytically. If you’re into expanding potentials using Fourier series, be my guest and read it through. In this case, I, however, prefer to look at what overall trends from the simulations tell us.
1 However, this is decidedly not the case for planets in P-type orbits. As demonstrated by Pierens & Nelson (2008), circumbinary planets with mass greater than that of Jupiter are likely to encounter instabilities during formation and will either be ejected or will migrate drastically, possibly ending up orbiting just one of the stars. That said, as noted in Quintana & Lissauer (2007), which I’ll discuss later, no terrestrial-mass planets have been detected so far in P-type orbits.
2 Keep in mind that the early reports of one or more planets in the system turned out to be incorrect; these are hypothetical planets.
3 Note that it's not impossible for both stars in the system to have protoplanetary disks, leading to planets in S-type orbits around each star.
4 μ is calculated as
where M* is the mass of the star around which planets formed and MC is the mass of the companion.