First, Some Notes:
- Your smaller planet is about 1/3 the size of our moon. We can keep calling it a small planet but some know it all would probably tell you it's a moon (exactly like I am doing right now).
- You mention the objects co-orbit, which is true of our Earth-Moon system as well. However the center of mass of the Earth-Moon system is inside the Earth, so viewed from the frame of the center of mass the motion is better described as the Earth wobbling as the moon moves around it. Your system would look the same, with even less of an Earth wobble.
- Your smaller planet probably wont re-form into a spherical mass, unless enough ejecta falls back to the surface. It is near the limit where a celestial body would start to become spherical but I think with the dimension you give it could be reasonable to say it might go either way.
The Answer: Sure, With a Caveat
This first depends on whether your smaller planet survives the breakup. Even though half to three quarters of it is intact, we don't know whether the catastrophe has altered its velocity. We do know its mass has been reduced by up to 50%. It is quite possible that the smaller planet's orbit has been altered enough that it will collide with the larger planet. This would be catastrophic for both, obviously. Depending on the nature of the catastrophe I guess it is also possible that the smaller planet will escape from the larger planet. Since your question is probably assuming the orbit of the smaller planet is relatively unaltered, let's go with that.
Assuming the Smaller Planet Stays in a Stable Orbit
When your smaller planet is split, the rocks and dust that are ejected will essentially each have their own orbit within the system. Any piece of ejecta can be considered a negligible mass object in a restricted three body system. There are six main outcomes for a given piece of ejecta:
- Remain in orbit around the smaller planet
- Collide with the smaller planet
- Fall into orbit around the larger planet
- Collide with the larger planet
- Orbit both planets
- Escape the system
In fact all outcomes will probably happen and the eventual distribution of ejecta falling into each category will depend on the nature of the catastrophe. We can have a guess at how this will look though.
Orbital Mechanics Ahead:
If you are only interested in what is technically possible, you may disregard the following and you will probably still end up with a believable scenario. If you want to go deeper and/or are interested in how each outcome may happen, read on.
Remember my mention of a three body system? Here is a plot of what that looks like:
Your larger planet is M1 and the smaller planet is M2. The five points are Lagrange points. The contours are an indication of the energy an orbit must have. Mathy details aside, we can imagine what might cause each outcome listed above:
- If ejecta has enough (but not too much) velocity in the right direction, it will enter orbit around the smaller body. If the catastrophe ejects material out into one plane, the ejecta might form a small ring around the smaller planet.
- If the ejecta is launched roughly outward without enough velocity to escape, it will fall back to the surface of the smaller planet. If the smaller planet has an atmosphere the drag will help more ejecta fall back to the surface. If the catastrophe created heat, the atmosphere will expand and the effect will be exacerbated.
- If ejecta is launched towards the larger planet with enough velocity it could enter orbit just around the larger planet.
- Same as the previous case, but ejecta is launched directly at the larger planet or hits enough of its atmosphere to create drag and fall to the surface.
- This is where the above image comes in. In a restricted three body system there are many possible orbits that move between the two main gravitating bodies. Here is a short example of a few possibilities. Many more simulations like this can be found using the google machine. Some ejecta will end up on these looping orbits, which may eventually settle into an orbit around one planet or the other. Also note the L4 and L5 points. These are "stable" equilibrium points around which orbits may form. After some time you will eventually find clouds of dust and rock at the L4 and L5 points ahead of and behind the smaller planet in its orbit. L1, L2, and L3 are "unstable" and thus ejecta won't accumulate there.
- Some ejecta may well have enough velocity to escape the entire system. At L2 is a "neck" through which ejecta has a higher chance of leaving the system due to the low energy required. If the catastrophe happens on the side of the smaller planet facing L2, much more ejecta will leave the system entirely.
Responding to Questions in Your Comment (revising size)
Sorry apparently I can't be succinct enough to do this in comments
- If you're making the smaller planet 1.5 times the size of the moon, the remaining 50-75% post-catastrophe will re-form into a spherical body.
- It is definitely possible for some ejecta to remain in orbit around the smaller planet. How much will depend on the nature of the catastrophe.
- The system could also be in geosynchronous orbit, but this is determined by its orbital radius (roughly 42,000 km for Earth). So if it were already geosynchronous and kept its orbit that would work. Note that geosynchronous means the orbital period is the same as larger planet's rotation period, but if the orbit of the smaller planet is inclined relative to the larger planet's rotation it will appear to make a figure eight across the northern and southern hemispheres. If you want it to be geostationary (sitting over one spot) it would have to orbit above the larger planet's equator at the geosynchronous radius.
- If your larger planet has the same rotation period as Earth, the smaller planet will look massive in the sky as it is 1.5 times bigger and 5 times closer than the moon currently. However, the slower your larger planet rotates, the further away geosynchronous orbit will be.