1. Very small, porous planet
Small mass, weak gravity. Need to be more like a spongy asteroid, but with small inhabitants, a small world can still be arbitrarily large and rich in proportion.
A disadvantage of small bodies, though, that they can't really have an atmosphere. You may not need one, but then make sure it rotates slowly, otherwise things lifting off would not fly up, but away.
For how exactly that lifting-off might work, just one example: thousands of small geysers erupting from the pores, when facing the sun, could shoot rocks up, and also provide your rain falling up.
2. Twin (double) planet
Gravitationally locked binary systems of even Earth-like size, in extreme proximity, are possible. These have a low-gravity zone in between. The idea of a binary planet has been explored e.g. in Robert Forward's Rocheworld, including the peculiarities of their gravity. E.g. the two bodies even share an atmosphere. A quote from the plot, quite in line with your up-falling water idea:
[the twin planet was] approaching a period where the configuration of
the star and planets of the system allow for a phenomenon where the
ocean on the water lobe of Rocheworld can partially flow to the rocky
lobe, due to the change in the gravitational equipotential.
3. Centrifugal, but safely below the escape treshold.
As pointed out by others, centrifugal force won't work, as soon as it actually balances out gravity. So, it shouldn't.
There could be an angular velocity, where the planet doesn't disintegrate yet, but most of its gravity is cancelled out at the equator.
This would make the planet highly unstable, magnifying the effect of all sorts of transients caused by environmental fluctuations, which couldn't make a difference in a slower-rotating case. These extremities may genertate some very useful chaos, with just about any surprising and unpredictable phenomena you could imagine.
Not sure about an atmosphere, though. Again, I can imagine that it could still hold one that's higly layered, with ring-like streams of matter (following up on the Roche-limit pointed out by @Laurel), possibly circulating back toward the poles.
The climate would certainly be hostile (e.g. due to the lateral speed differences between layers), with thick, hot, whirling winds of smoky-dusty matter blown off the surface, but may perhaps help your creatures gradually form their own half-natural "planetary Dyson sphere" (actually: a belt) out of debris: a shell, or shield in the sky they may inhabit and maintain (e.g. because the weather would kill them on the surface).
Examples of effects lifting things off:
Solar winds, ionized clouds generating strong electrostatic fields (e.g., periodically, synced to the orbit), attracting oppositely charged chalky, porous, light minerals with peculiar electrical properties (constant, heavy dust storms due to the fast rotation can charge objects on the surface, as well as polish them to pebbles). Then the flying pieces would eventually discharge up there, and fall back down, closing the cycle (-> your replenishment requirement), while also providing energy to your sky-dwellers as a bonus.
As already described, at the boundaries of centrifugally separated layers/rings of floating matter all sorts of interactions could really spice up the weather. Winds can lift things up even here, on Earth, and warm air can do it, too. In your world, with much less pull from gravity, strong currents could blowing vertically upward and easily carry small objects.
Whirling ionized matter in that thick, layered atmosphere, could also generate high electric currents (e.g. thunderbolts) and thus strong magnetic fields (fitting that other aspects of your story, too), lifting certain metallic rocks, as others have already mentioned.
It may even be some partly artificial effect (depending on your story): the guys in the sky may exploit and control the instabilities, deliberately triggering certain features of the climate, precisely to get some stuff up there.