I was wondering what would happen if there was a planet with a "shell" of stone entirely surrounding it (not connected, just held in place by gravity), with a few holes all the way through the outer shell, and two civilizations - one on the outside, and one on the inside. Would this be feasible? Would the outer shell need to be made of a specific material for it to work? Besides the fact that the outer civilization would likely suffer from droughts (as all the water flows into the interior), what consequences could this have on the two civilizations?
This sounds very much like a naturally-occurring Dyson Sphere that you're describing, albeit around a planet rather than a star.
Such a structure would not be able to form naturally at the birth of the planet, as planet formation pretty much consists of a bunch of stuff smashing together until it all smooshes (that's a technical term!) itself into one big ball. The only way to wrap a "core" with a "shell" would be if the shell formed entirely at one instant in time, otherwise the pieces would simply be pulled down to the core as it started to form, and would never have the time to form a shell.
The other option would be erosion, underground water eating away at the subsurface material to form a ginormous cavern that eventually encompasses the entirety of the underground. However, this, too, is so unlikely as to be implausible beyond belief; among other problems, where would the eroded material go? You'd have to put it somewhere, and unlike the water itself which could evaporate and find itself back up on the surface, the only direction it can go is down. You'll also quickly end up with caverns that simply wind their way down, down, down, down, and while they might link up with others the water's no longer eating away on the stuff that's still hanging out up because, of course, water flows downward.
Even if you did somehow manage to create this structure, it would be incredibly unstable -- one good break of a piece of the outer shell (caused e.g. by erosion weakening the integrity of the shell until it just breaks under its own weight) and now you have an unstable mass that will simply crash down to the core, break up, and now you're back to a solid planet.
And, of course, even without the aforementioned issues, even a small amount of tidal forces exerted on the planet would "wobble" the shell until it crashed down into the core. (EDIT: To say nothing of the sheer devastation should a meteor of even moderate size impact the shell!)
If we put aside such questions, though, and assume that we could have it (artificially-created by an advanced civilization, magic, etc.), then the effects depend largely on its structure.
For example, the surface might be quite dry, as you suggest, if there are enough holes to drain the water but not enough for ample circulation to bring water vapor back up to form clouds and rain. Or, if you've got ample air circulation between surface and subsurface, you could have plenty of moisture up top with ample rains for crops and such to thrive; this wouldn't even require the whole thing to be swiss cheese either, just a few, say, lake-sized holes. You could even help the surface out by having the shell be thick enough to support a few lakes of its own; not to mention that mountaintop glaciers can feed rivers that further add to the surface's water availability.
The lower layer would likely be a dark place, though if you have the circulation to keep the surface non-drought then it would most likely not be the damp and dank environment you see in caves. If you have lake-sized holes on the surface, you'd have lake-sized patches of ground in the underground where they'd get enough sunlight to grow crops; otherwise they'd be reliant on non-photosynthesizing food sources, such as fungus and many types of "slime" (generally algae). (Note that these lake-sized oases of sunlight wouldn't be directly below the holes unless they were precisely on the equator, and even those would of course move around with the seasons -- and in fact this movement might make all but the tiniest fraction of the space worthless for all but the fastest-producing light-dependent crops.) Given enough space between the subsurface's ground and the ceiling, they'd have their own weather patterns as well; with enough and large enough holes in the shell, the would be influenced by (and influence in turn) the surface's weather as well.
The most likely consequence is that you'd have two wildly different civilizations. The surface dwellers would likely view the "underworld" as a dark, scary, foodless desert, while the subsurface dwellers would likely view the surface as a blindingly hot wasteland -- neither would find much recognizable food in the other civilization's demesne, and both would find the other's climate intolerable to their senses in almost every aspect. I'd expect this to feed into rampant distrust of one's opposite civilization, and most likely there'd be wild rumors about the "barbaric" practices of the other; with two civilizations so wildly different, these rumors would likely be grounded in some element of truth, albeit embellished to varying degrees. (This assumes they're even aware of each other and have any sort of interaction at all, which may not be the case -- the vast physical gulf might be enough to keep the two from ever knowing about each other, at least until someone finally figures out how to make a rope long enough that won't snap under its own weight...)
Apart from the problems of formation and the problems of mechanical stability already mentioned by others, there's also the problem that gravitation cannot preserve that situation.
If the sphere were perfectly symmetric, then the sphere would neither feel a net gravitation from the planet, nor the other way round. Of course the shell would feel the planet's gravitation as stresses, but those would not affect the movement. The planet would not even notice the sphere.
If the shell were absolutely tight, one might speculate that the air is denser in the inside of the shell than on the outside, and thus the atmosphere in between would keep the two spheres concentric (since air pressure gets less with height, whenever one side of the shell gets closer to the planet, it gets larger pressure and thus gets pushed back; I admit I didn't calculate whether this really works out that way). However since your shell has holes, any difference between the inside and outside air would quickly equilibrate.
One thing that might work in theory is if the outer shell is superconducting, and is held in place by the magnetic field of the inner planet. However you'd have to assume that there's a material that's superconducting at the planet's temperature. Also, a shell out of such a material would probably not happen naturally; it therefore would need to be a constructed object.
(I am assuming you are asking this because you are interested in how something like this could exist within physics similar to what we have in our world, rather than an imaginary universe.)
I don't think there's any way that would just happen, in the physics of our own world, anyway. It's not a stable form that would be held in place by gravity, as you say. It would need astronomically powerful technology or magic to create it and hold it together. No sphere of solid matter the size of a planet would just hang steadily above a lower one without being insanely strong or somehow floating... maybe there's a massive magnetic field from the core and the outer shell is mostly magnetic metal suspended somehow - but this seems astronomically unlikely to me, unless the physics of this world are very very different from ours, or some amazing magic or technology explains it. Not that authors haven't written about such worlds in fantasy or extreme sci fi: see Ring World, for example.
The problem you mention with droughts on the outer shell sounds much less likely to be an issue than other concerns, and could be easily avoided by the creators if they so chose, via ridges around most or all of the holes, for instance. Also, depending on what the atmosphere was like and the altitudes involved, evaporation could bring water out to the outer layer via rain. But if the force keeping the outer shell suspended also acts on water, then water too would tend to just pool in a floating layer rather than falling down to the inner sphere. Unless the force is magnetic as so doesn't act on water.
Apart from the whole thing collapsing, the main effect on people living in such a situation that I see, is that the lower world would be extremely dark and so probably have a hard time growing earthlike plants. But any magic or tech or other-universe effects that explain the hovering shell, could more plausibly explain away such conditions or problems.
Depending on how the forces work that keep the outer shell hanging there, I can also imagine that someone flying high enough off the surface of the lower shell, might tend to start falling up towards the inside of the outer shell - it might have a force pulling things up to it too, leading to a third environment. If that is the case, then someone might be able to hang-glide or parachute or baloon-ride into a hole and make it to the other side of the outer shell. If the shell suspending force is magnetic or doesn't work that way somehow, then people might be able to similarly travel down relatively easily.
Depending on how the forces act on the atmosphere, the air pressure might also tend to be massively greater on the bottom rather than the top. Consider that on Earth, the tops of the highest mountains have air so thin that it's barely possible to survive. The explanation for how the suspension force works would do well do consider what it does to the air.
There would probably be some really strong winds and weather patterns around the holes, especially on the outer shell. If oceans are draining into the holes, they would of course also cause periodic dumps of water into the atmosphere of the inner planet. Again, what happens to that water would depend a lot upon how the suspension forces and atmosphere work.
The astronomy and astrology of civilizations on the inner planet would of course be based on the layout and movements of the holes in the outer shell.
A natural system by which a planet is enclosed in a shell seem possible, but very unlikely. That means that there are millions of them in the universe!
Consider a planet with many moons at the same mean orbital altitude but differing orbital planes. Collisions could cause a natural Kessler syndrome. One could postulate a world where the debris coalesced, especially if it is relatively on the outside edge of the habitual zone and there are many icy volatiles present. Remember, water expands as it freezes.
Now, how did the original moons come about? Perhaps a gas giant is gravitationally cleaning out the Oort cloud. Thus, you would have a source for the moons and a source for more icy volatiles.
The only real similar situation that comes to mind here is the planet saturn. It looks a bit like a sphere around a sphere, except that the outer sphere has 2 (big) holes, which make it look like a ring.
If you were to build something around a planet, you would have to start with a planet like saturn, then try to redesign the ring to be more in a shape that you like, and see how far you can push the change before it becomes unrealistic.
Just a note: I would expect you to be able to design a sphere big enough to build on, but you probably have to build something like a dome dome due to the lack of atmosphere.
Maybe it's not natural, but created by an earlier civilization before the current occupents evolved.
Building on another idea posted in this thread, the inner could be the original planet and a "shield" was added around it. That got covered in regotith over geologic time with nobody cleaning up; perhaps a downed moon added substantially to the material. Primitve live spread to the newly available zone and it evolved into a whole new ecosystem.
There is a concept of a shellworld discussed in various places you can easily find on the internet. To address the question of "Would this be feasible?", my answer for shellworlds is that they are, but only for certain configurations. I went into the specific mathematics here:
It would technically be possible for people to live on both the outside surface of a shellworld, as well as on the surface of the planet inside it. However, it's difficult to make this work with both levels still having an atmosphere (although I won't go so-far as to say impossible). It is impossible to make work with holes in the shell. The levitation of the shell is dependent on the air pressure underneath it, as well as the ability of the shell to hold the pressure barrier. You would need air locks to go from the outside to the inside.
The stability requirement isn't hard to satisfy, provided the planet is sufficiently large. However, the larger the planet, the harder it is to maintain a perfect pressure barrier over such a large surface. So pick your poison.