So my story is set in the TRAPPIST-1-system and a rogue planet enters the system. When it's still far off, a scientist discovers that one of the numbers is slightly off. (She has some tables that predict where the planets are going to be since the entire system is a periodic solution to the n-body-problem.) I figured that the effects of this would be much more extreme in such a small system since all the planetar bodies are so close to each other, rather than in our solar system where a planet could just pass through without doing much damage. So what would happen if it just got out of balance a tiny little bit?
This question asks for hard science. All answers to this question should be backed up by equations, empirical evidence, scientific papers, other citations, etc. Answers that do not satisfy this requirement might be removed. See the tag description for more information.
The thing about planetary systems - and many $N$-body systems in general - is that they are fundamentally chaotic. That is, small changes grow over time, eventually creating wildly divergent results. A way to quantify this is the Lyapunov exponent $\lambda$ and the Lyapunov time, $\tau=1/\lambda$, which gives us an idea of how quickly small changes grow. For the Solar System, $\tau\approx5$ million years (Laskar 1989), which is small, in terms of the evolution of most planetary systems, and over the course of about 100 million years, an initial change in Earth's orbit of 15 meters can grow to an uncertainty of 1 AU. In short, even small perturbations to planetary systems can render them wildly unstable, and studying - and even verifying - the stability of the Solar System is an important field of study.
The TRAPPIST-1 system is dense, with all seven planets within about 0.06 AU of their parent star (Gillon et al. 2017). How can such an arrangement remain stable for so long? The answer is resonances, where the orbital periods of the planets are related by ratios of (in this case small) integers. The seven planets fall into near-resonances with one another that ensure stability, brought about via migration through the protoplanetary disk (Tamayo et al. 2017). Now, while it's been found that a wide range of initial conditions around the time of planet formation lead to resonances in the system, deviation from these resonances can easily lead to instabilities. As Gillon et al. wrote of their simulations,
We investigated the long-term evolution of the TRAPPIST-1 system using two N-body integration packages: Mercury and WHFAST. We started from the orbital solution produced in Table 1, and integrated over 0.5 Myr. This corresponds to roughly 100 million orbits for planet b. We repeated this procedure by sampling a number of solutions within the 1-$\sigma$ intervals of confidence. Most integrations resulted in the disruption of the system on a 0.5 Myr timescale.
We then decided to employ a statistical method yielding the probability for a system to be stable for a given period of time, based on the planets' mutual separations. Using the masses and semi-major axes in Table 1, we calculated the separations between all adjacent pairs of planets in units of their mutual Hill spheres. We found an average separation of 10.5 $\pm$ 1.9 (excluding planet h), where the uncertainty is the rms of the six mutual separations. We computed that TRAPPIST-1 has a 25% chance of suffering an instability over 1 Myr, and 8.1% to survive over 1 Gyr, in line with our N-body integrations.
Observations indicated that the system's arrangement cannot change significantly without drastically increasing the chance of a catastrophic instability. I'm fairly pessimistic of the chances of the system to survive in cases of most close encounters, although there might be a line to be drawn. For instance, an object the mass of Mercury passing at $\sim$1000 AU would give better odds for stability than an object the mass of Jupiter passing at $\sim$1 AU.