I was wondering if three planets can rotate around each other so that:
Two of the planets are smaller and rotate around each other
Both of the smaller planets rotate around another, bigger planet.
It's a bit hard to explain, so I made a (low-quality, just illustrative, using Mercury, Venus and Earth like planets) animation using GIMP and Blender:
Is this possible (not exactly like in the animation, of course)?
In theory, yes, this is possible. In practice, it would be a rare thing to encounter, just because the laws of nature can be fickle and somewhat unpredictable. I'm going to call the planet your animation shows as Earth as PB1 (Planetary Body 1). I'm also going to call the Mars one PB2, and the Moon one PB3.
There are some important bits, though:
You can fool with a planetary orbit simulator, like the one here, or here, or here, to find out how difficult it is to set up. (It's even harder to set up, say, a horseshoe orbit.)
Cem Kalyoncu messed around with the second simulator, and found some nice settings for your planetary system:
PB1: Mass: 200, Location: 150, 0, Speed: 0, 133
PB2: Mass: 100, Location: -100, 0, Speed: 0, -105
PB3: Mass: 100, Location: -50, 0, Speed: 0, 105
Thanks Cem Kalyoncu!
This is totally plausible, albeit with much larger separations than you show in the animation. You can easily have two bodies orbiting each other in space. Now, take one of those bodies and turn it into two bodies on a much smaller orbit. This works as long as the two smaller bodies are close enough that their mutual gravity is much stronger than the gravity of the other, distant body. (In technical terms, this means that the binary planet has to remain within the Hill sphere).
The punchline: there are plenty of little technical issues like tides, but this is totally plausible in principle.
It's basically analogous to multiple star systems, which tend to be set up in a hierarchical fashion, where any star is much closer to its closest companion than to any other star. If you're interested, I created a couple of fake systems in this mold. See here and here.
It's pretty easy to conceptually convince yourself that is possible. Just replace the biggest planet with the sun, the medium one with Earth, and the smallest one with the Moon and you have the same setup.
There is nothing special about stars, planets and moons as it applies to the way things orbit each other. It's just that the two smaller bodies wouldn't be planets by definition because they orbit another planet. They're both moons.
Your scenario is not super likely, because in order for the smaller bodies to stably orbit each other, they have to be sufficiently far away from the larger planet. But that makes them less strongly tied to it and more likely to be disturbed by something else like a nearby planet or just the central star.
Using the second simulator mentioned in the accepted answer, I came up with this:
This setup was like the OP wanted, and was stable for so long my computer ran out of battery, even when switched to the fastest mode.
Playing around with the sim suggested above, I got the configuration desired with the following:
Body 1 - Mass 200, position: x=0, y=0, velocity: x=0, y=-1
Body 2 - Mass 50, position: x=110, y=0, velocity: x=0, y=140
Body 3 - Mass 0.001, position x=125, y=0, velocity: x=0, y=305
This was stable for a very, very long time. Should be able to get this to orbit around a very, very massive star without too much headaches.
body[3].velocity.y should be 31 instead of 305 to make it work.
$\endgroup$
Well, yes.
For example, our Sun, Earth and Moon.
I'm not sure if this is exactly what you're going for, but have a look at how Pluto and Charon interact with Pluto's moons:
