In D&D Spelljammer, there's a concept of "crystal spheres" that enclose a solar system, with its radius being double the radius between the sun and its furthest planet. For my solar system, I did all the calculations for its "habitable" zone, and I found out that I can (barely) fit two habitable planets within it. I was hoping for several unique planets within the solar system. My question is, if the entire solar system was enclosed, would the sun's heat be trapped, almost like an oven? I think it would radiate slowly through the crystal, but would the ambient heat be enough to theoretically extend the habitable zone?
Yes, the habitable zone would be extended. And widened as well. This is because planets would gain some heat from infra-red radiation of the sphere and it would be uniform inside the sphere instead of centered on the sun. How big the difference is depends on the inner surface temperature of the sphere, which in turn depends on its ability to radiate heat outside. Note that this also evens out the seasons!
Reflectivity affects the spectrum the sphere radiates at. A more reflective sphere will contribute larger proportion of its energy in the visible spectrum. The night would not necessarily be dark.
And other effects are possible. A sphere that is reflective in the UV range would make seeing UV light much more practical. More likely it would be set to reflect in the near infra-red and night vision of many races would be boosted by that.
But you should ignore all that. Spelljammer has as its basic premise that pretty much all that we know about space does NOT apply. Specifically the spheres, the suns and the planets are created with and kept in order with divine magic. So the habitability of a planet is whatever who created it set it to be. Unless some other divine class power meddles with it or the powers that set it up lose their power or some similar stuff.
While you could mix science with it and stuff like there is more near infra-red because of the sphere and the sky looks very different to elves than it does to humans is a nice touch IMHO, that is probably as far as you should take it. Specifically that the celestial mechanics and planetary formation and habitability do not follow scientific principles is, as I said before, part of the main premise of Spelljammer. It is connected to the main point of doing Spelljammer at all. If you want your setting to be like Spelljammer, you should keep that unscientific IMHO.
This does actually apply to some extent even when the sphere is actually a precursor Dyson sphere. The precursors would have shaped the way the sphere radiates to affect the planets in the way wanted. If you are going to build a Dyson sphere and keep habitable planets inside it taking that extra step is absolutely trivial. Amusingly this would allow different planets in the same system to see different constellations or the constellations to actually change over time because the stars would actually be simple sources of light pointed at the planet covering the sphere. The administrative AI would have full real time control of what people see.