I have a planetary system with a yellow dwarf similar to our own Sun and three habitable planets. I want to hide this system from sight and long range scans in a nebula, a giant cloud of dust and gas in space. Can this planetary system exist safely hidden within the nebula or within a pocket of empty space in the nebula?
We have plenty of examples where stars have been hidden by nebulae - and not just newborn stars. Typically, the gas and dust comes from mass loss from one of the stars in the system. Examples include
- LL Pegasi, a binary system containing a carbon star that is sloughing off large amounts of dust as it nears the end of its life. This makes the star visible only in infrared light.
- CW Leonis, a similar star which seems to have a binary companion that has evaded direct detection; its existence was only discovered through measurements of the motion of the primary.
Some stars, as L.Dutch indicated, can ionize the gas surrounding them, producing HII regions and emission nebulae which are easy to find. However, these stars are typically hot and massive; their high temperatures mean their emission peaks at shorter wavelengths, and therefore they emit more high-energy photons capable of ionizing the circumstellar hydrogen. Your star, on the other hand, should be fine, as it's comparatively cool.
A yellow dwarf seems comparatively unlikely to form a dust cloud while it's on the main sequence; later in life, as it enters the asymptotic giant branch phase, it could if it indeed becomes a carbon star like the stars I mentioned above. To form this dust cloud, then, perhaps there's a companion stars on an eccentric, long-period orbit, constantly replenishing a large circumstellar cloud of dust that enshrouds the system.
(Of course, as Juraj suggested, the system could simply be moving temporarily through an interstellar cloud, e.g. something as dense as a Bok globule. I'd been thinking largely of systems that would be stable over longer periods of time, and I'd completely missed that possibility.)
It's true that the dust will reemit light, peaking at a few microns, firmly in the infrared part of the electromagnetic spectrum. CW Leonis, for instance, is extremely bright, as seen from Earth, at 5 microns. But unless the people doing the scanning are carrying instruments capable of imaging the system at those wavelengths, it won't pick anything up. If they're on a spaceship, I think it's unlikely a ship would have such an infrared imaging system; most objects radiate at plenty of other wavelengths, and carrying optical or radio instruments seems much more efficient.
When you want to hide a star, distance is your friend. Due to the famous law, the luminosity of a star decreases with the square of the distance. Therefore the farther the observer is, the more difficult it is to see the star.
But what happens if you put the star in a nebula?
The light of the star will excite the atoms in the nebula, getting them to emit light. As a consequence, instead of a dim point, you have now a diffuse glow that shouts, for those who can hear, "hey, I am hiding a star here!".
Wrapping up: if you want to hide the planets, maybe the nebula is a good idea, as the current technologies we have for detecting planet rely on the direct observation of the emission of the main star. If you want to hide the main star, better rely on attenuation with distance.
If you are worried about the habitability of these planets with a nebula, they could be just fine. As star systems travel through space they can pass through nebulae of varying sizes and have nebulae form near them and fill the sky with colour. I imagine the night sky on these worlds could be spectacular.