Numerical Modelling Is The Way
As stated in comments, the three body problem is an unsolved problem in physics, which includes all two-moons-and-planet system. No "nice" equation exists, although the numerical model method give you an answer with some "nice" concepts and a thing to measure.
The basic approach is to figure out the six gravitational forces in the system, apply those forces to each body (moon/planet) over a small time step to get new positions, and repeat.
Until the heat death of the universe, the power goes out, the computer breaks, things crash into each other (in your model or otherwise), or you get tired of it.
Where is the payoff? Well, you count how many steps until one of those above conditions happen in/to your model, and that is how you measure stability.
There are a lot of little questions when doing this that get the pendants really going. Like... is your time step short enough? Did you check for collisions? Did you check for bodies passing through each other? Did you go long enough? Did you account for any liquid rock which may be going on? Did you model them anywhere along the sliding scale from three solid bodies to every atom in each body? (They will make uncomfortable noises no matter what you say: either uncomfortable for you or uncomfortable for them.) What if each body has a magnetic field? And it goes on...