The planet I'm working with has two different moons, both around the same size (3/4ths that of the main planet). Is it possible for the planet to support the orbit of these moons, especially with these proportions?
Yes, it's possible
However, you may have to widen your definition of "orbit" somewhat!
Usually it's relatively safe to say "the moon orbits around the planet", which leads most people to visualize orbits like this:
Fig 1. (As with most diagrams like this, it's not to scale; the moon would be much farther away! But then it'd be very small on the diagram.)
The light blue arc is the planet's orbit around the star. The pink circle is the moon's orbit around the planet.
With Earth's moon, the above diagram would tell the tale (more or less), because our moon is much less massive than the Earth.
However, with a moon that is about half the mass of the planet, the gravitational pull of the moon on the planet becomes significant, and this is what you would see instead:
Fig 2. While the "planet" and "moon" are (still) not to scale, the inner/outer orbit circles are approximately to scale! That means the planet would be heavily "yanked" around by the moon. The planet and moon orbit a common center point known as the "barycenter", and they both then orbit the star (light blue line, as before).
This is known as a double body system, but for our discussion, I'll stick with "planet" (or "primary") and "moons" (or "satellites").
Fig #2 is the most basic way to arrange the orbit. As with any system, the orbits could be elliptical, and/or eccentric (where the ellipses overlap each other, like my 2nd diagram in this question about binary stars, which follow the same basic rules.)
I assumed size = volume, which means the moon has about 42% of the mass of the planet. If you instead take size = mass, so the moon has 75% of the mass, it makes the planet's orbit even wider, but the overall shape is the same.
Adding in the second moon
So far I've only addressed the question of one massive moon. Pretty much everything I've said above applies equally to the 2nd moon. But the orbits get more complicated, as the 3rd moon can start interfering with the orbits of the other two objects (or vice-versa). The orbits get even more complicated if they are eccentric (see above).
For example, the objects can now pull each other into higher orbits or even eject one object from the system, sending it out of its (satellite) orbit and (likely) into its own orbit around the star.
Collisions aren't out of the question, although they're rather unlikely.
How would they form?
One of a few possibilities:
- Leftover swirling "dust" from the creation of the solar system forms the moon.
- Large proto-planet (mass nearly as big as the planet) collides with the planet and knocks a huge mass of debris into orbit that is then pulled together by gravity into a round moon.
- The moon could be a rogue planetoid, captured by the gravity of the planet
Play Simulation Time!
If you'd like to see something like this in action and have a chance to play with the parameters, try UNL's eclipsing binary simulator. It's made for binary star systems, but the principles work just as well for planets and moons. I don't know of any online orbital simulators which can handle more than two bodies (to anyone reading: please leave a comment if you know of one!), however even this two-body simulator could be quite instructive!
Your question asked about the "required proportions" to make this work. Hopefully this question has already answered that, but I realized I didn't actually address the point directly. As long as you don't have the objects so close that their atmospheres (or surfaces...) collide, the only thing you have to worry about are potentially strong tides (not only liquid ocean tides (if you have oceans), but the same tidal forces could cause seismic tremors if the planetoids are rather close together).
Someone may happen by and mention the Roche limit, which is the radius within which one of your moons could actually be ripped apart by tidal forces. You would have to design quite an extreme system indeed to have to worry about the Roche limit, assuming your planetoids are all of a typical solid, rocky construction.
So, back to the question of "required proportions": as long as you keep the objects reasonably distant (say, on the order of the Earth-Moon distance of about 385,000 km), with Earth-like masses, you should be OK. I'd suggest you use the above simulator to help you visualize what your parameters might look like.
The only examples that we have of multi moon systems is where either the multiple moons are very small relative to the planet or where a smaller moon is a much greater distance away than the larger moon. Two large moons would likely perturb each other enough to kick at least one of them out. It is very unlikely that they would chance into any of the very few (if any) stable systems.
Now if one large moon orbited close and the other orbited far enough away to treat the planet and inner moon as a single gravitational source, it might work as well.
In any case, if you have a moon that is 3/4 the size of its planet, you pretty much have a binary planet.