If you have several stars, you can have lots of wacky trajectories - including orbits around empty spaces (where some interesting gravitational point is located).
Now, there's a problem with designing stable many-stars system, but there's a way to "solve" it:
Controlled singularities (not black holes)
If civilisation you have at hand vastly exceeds ours at math, they could've solved something useful for n-body problem.
Usually our mathematical objects behave nicely - you ask them what would happen at time T and they answer. However, there are some problematic points where math is stretched too much or simply broken so you can not really predict what would happen after that point. Those problematic points are called singularities.
For example, if you had ideal rigid needle and ideal rigid billiard ball launched straight at it tip, little deviation in ball's position or direction may result in huge change of its trajectory after the collision.
Bad thing about n-body problem is that they have those.
Good thing about that is that you can exploit those singularities if they're not singularities for your plot-powered math. If one of system's suns is going to deviate too much and break everything, there could be some smaller deviation (like, planet's movement) that would be amplified to counteract sun trying to go off the leash.
On the other hand, it may happen that there does not exist smaller deviation to prevent some big deviation but your players would have hard time proving that, even if they try to.
Planet, in turn, can be nudged by something yet weaker - so that this nudge would be amplified at another "solved singularity". This nudge could be achieved through some atmospheric happening (which changes tidal picture a bit, which nudges the moon a bit, which nudges the planet a bit).
Atmospheric effects are ripe with similar stuff (singularities) too. The same means could be applied here to make everything work through weather control stations at several "balance" planets - uninhabitable planets that are mostly used for computing desired corrections and adjusting weather at specific points in a specific way to achieve that.
Think of it as employing a butterfly and super-super-computer to tell it exactly where and when should it flap its wings to divert a typhoon from densely-populated area to somewhere of lesser significance.
Of course, if those folks are exceptionally good they may be able to design planets with atmosphere that automatically reacts to changes in distances to suns with controlling nudges. However, that is harder and there would still be need for adjustments for stuff they didn't account for.
For example, there can be some rare-visiting comets, or some rogue celestial bodies. It's easier to spot those things from far away and account for them with total-wreck-preventing nudges, than secure a space where won't be any.
Also, for certain trajectories and/or masses of rogue celestial bodies it may be impossible to counteract effects of their passing, but that doesn't really make the system unstable.