There are several questions around regarding stealth in space, where the general conclusion is that stealth is not really feasible because (among other things) heat radiation from a spaceship doing anything interesting (except unmanned probes just coasting on a fixed trajectory) can be detected from millions to billions of km away.
One part of these answers is that high power rocket engines emit large amounts of heat radiation, so stealthy maneuvering is limited to e.g. cold gas thrusters or mass drivers, which have a low specific impulse. However that reasoning seems to assume current rocket engines that are not designed with stealth in mind. Rocket engines themselves get very hot, and the exhaust plume is very hot as well, so both emit massive amounts of thermal radiation.
Shrouded rocket engine
One item I have not seen answered in anything I have read is the following scenario:
Assume that only detection from one direction is relevant. If the heat signature of the ship and rocket engine can be hidden from one hemisphere of space (i.e. a 180° "cone") that is good enough. We will assume hostile sensor platforms are not all around the ship's location.
The ship uses large radiation reflectors (think JWST sunshield) to block visibility of the rocket engines themselves and the first few meters of expanding exhaust gas from the stealth direction. As rocket exhaust in vacuum expands rapidly, the exhaust plume quickly escapes the cover of the radiation reflectors, but at that point it has already expanded a lot. Note that the direction of firing need not be straight away from the enemy observation direction but can be up to almost a straight angle.
The rocket exhaust contains only gaseous species. Specifically, the rocket fuel does not contain carbon so there are no soot particles in the exhaust. (If soot-free combustion of hydrocarbon rocket fuel when run sufficiently oxidiser rich is possible, that is also an option. Current rocket engines run fuel rich.)
The exhaust gas that emerges from the cover of the radiation reflectors should then no longer emit much—if any—electromagnetic radiation, as far as I know, so this setup should allow a spaceship to remain stealthy (in one direction) while still using powerful rocket engines.
Details
Hot materials generate thermal radiation from collisions on the molecular scale, which then generate exited states in the molecules. When these exited states relax that releases thermal radiation. I would think that rocket exhaust gas in vacuum quickly expands to a pressure where there is practically no more interaction between gas molecules. Therefore the molecules can not get excited and as soon as any existing excited molecules have relaxed back to their ground state, the gas no longer emits any thermal radiation.
Soot (or other solid) particles in an exhaust plume can continue to emit thermal radiation until they have cooled down, and will also reflect sunlight. So practical rocket propellants to choose from will consist of hydrogen, oxygen, nitrogen and maybe fluorine atoms. The exhaust plume thus also consists of molecules composed of these atoms, which are all gaseous as far as I know. Assuming hydrogen and oxygen are used, there will be hydrogen gas and water vapor in the exhaust, and if nitrogenous fuels are used also nitrogen gas and nitrogen oxides. Hydrogen propellant from a NERVA-style nuclear rocket is also an option.
Fun fact: The ʻOumuamua interstellar object that passed through the solar system in 2017 exhibited non-gravitational acceleration, some explanations for that involve outgassing of nitrogen or hydrogen which would not have been detectable with Earth telescopes.
There may be ways to still detect such shrouded exhaust plumes, e.g. some gases can fluoresce from solar UV radiation, or some gases might have some phosphorescent states that I am not aware of that will cause them to emit more radiation when they have already left the radiation reflector cover. But I am hoping that if such phenomena occur, the visibility of an engine burn will still be reduced by several orders of magnitude compared to a non-stealthy engine burn.
I don't think detection of the exhaust plume by analyzing starlight from background stars would be a major issue. Analyzing the spectrum of starlight requires a lot more photons and thus observation time than just detecting regular thermal emissions, so the detector needs to get lucky to have a bright enough background star at just the right position before the exhaust plume expands too much to be detectable. Even if that happens for one observation post, that would not allow triangulating the distance to the spacecraft and finding its position or vector of movement.
This question does not deal with other ways of detecting a spacecraft, e.g. radar or reflected sunlight, but from what I have read at longer ranges (say, 100.000+ km) the main detection method that can not be prevented is thermal radiation emission.
Question
What I am wondering about is if this scheme would be an effective way to allow a spacecraft to maneuver while remaining stealthy. The scenario makes quite a bit of assumptions, so please tell me if you think any of them are wrong, but the main question I would like to see answered is how easily detectable the expanding exhaust plume would be compared to the detection of a non-thermally-shielded rocket engine.
In order to have a properly defined question I will add some additional assumptions intended to establish a baseline. I hope the answers generalize to different situations, so please do mention if an answer sensitively depends on these details, in which case I may need to revise them.
In order to set some baseline, let's assume today's technology level in sensors. In a scenario with spaceships flying around detection of both hot rocket engines and expanding exhaust plumes will be better, but without knowing which will improve more compared to the other, today's technology is a good enough baseline. Putting stuff in space is probably a lot cheaper than today, so telescopes can be bigger and more of them.
Any type of rocket engine is fair game. The ship is trying to stay stealthy so pick the one that is the hardest to detect. As explained above I think that excludes anything running on a carbonaceous propellant. I think hydrogen or hydrazine with oxygen would be the most promising rocket fuel types. Nuclear engines are also ok if you think they would be harder to detect. I do want the ship to be able to go somewhere rapidly so assume a thrust of at least 1 meganewton (approximately the SpaceX Merlin).
Of course anything can be detected if you have a large enough telescope and enough observation time. So the question comes down to a ratio: How much harder would it be to detect a thermally unshielded rocket engine out of the background from one with thermal reflectors as described.
My guess is that there are at least three orders of magnitude of difference between them, which would translate to a ${\sqrt{1000} \approxeq 31}$ times smaller detection range. So I guess that a telescope optimized to detect unshielded rocket engines would be able to detect the chosen rocket without thermal reflectors 31 times as far away as a same sized telescope optimized for detecting thermally shielded rocket engines which is trying to detect that same rocket with thermal reflectors in place, given the same amount of observation time for each telescope.
A telescope designed to detect expanding exhaust plumes may need to observe different frequencies of electromagnetic radiation from an infrared telescope trying to detect hot rocket engines, but there doesn't seem to be an order of magnitude difference in the efficiencies of infrared telescopes compared to optical or other wavelengths, so it appears that setting the telescope size (aperture size) of both telescopes the same would make for a good way to compare them.
The best answer would thus be the answer that is able to nail down the above ratio as much as possible, though I don't expect more than an order-of-magnitudes answer. Or, of course, an answer that can show that one of my assumptions is wrong in such a way that the question is invalidated.
Addendums
Some answers question if gasses still emit thermal radiation when the pressure is low enough that molecules are no longer interacting. As an illustration I will link this Nasa video, which shows icicles forming on the engine bell of a hydrogen-oxygen engine. The engine bell itself is cooled by liquid hydrogen, but the icicles extend quite a bit away from the bell. The water vapor exhaust does not contain any soot particles and the water vapor and hydrogen itself emits little enough radiation that it does not melt the icicles sitting centimeters away. (The test is a throttle test, but the icicles don't melt when the engine is throttled up to 100% again.) As another reference, here's Scott Manley's interpretation of that video. The exhaust gas here is still at 1 atmosphere of pressure, but even in those conditions pure gases emit far less heat radiation than solid matter (like soot) because only a limited number of excited states are available in the molecules.
As for a JWST style sunshield not being able to cope with the radiation output of a big engine, the above also disqualifies that idea. Perhaps the emissions from the engine itself require some more durable shielding if the engine is not cooled cryogenically, but the exhaust plume should not pose much of a problem. And if so, the JWST sunshield consists of several layers of aluminized polyamide foil, it would be possible to replace the inner layers of foil by something that can withstand some heat like aluminium foil. Of course the exhaust plume should not impinge on the radiation reflector, if that happens all bets are indeed off.
From here, "Atmospheric Ultraviolet Remote Sensing", Robert E. Huffman, in International Geophysics, 1992, chapter 5:
Cold gas thrusters are used on many spacecraft to change the attitude of vehicles in pitch, yaw, and roll. This cold gas, usually nitrogen, cools and condenses into particles near the thruster. The particle cloud is an excellent scatterer of sunlight, which has lead to unwanted interference in measurements. An account of this scattering by thrusters using argon gas and its elimination by the use of neon for the thruster gas is given by Kolb et al., 1983, 1985. The locations of all thrusters on the spacecraft should be known and planned for by principal investigators.
So it seems that nitrogen cold gas thrusters are not as stealthy as I had expected. I'll need to figure out if this also applies to hot rocket exhaust and water vapor.
