This is roughly the same as Bald Bear's (first) answer, but more fleshed out...
The "ruins" are built (and removed) by "cloaked" repair mechs of unknown origin. These mechs have some form of probably anti-gravity propulsion (that is, they "float" in air and are near-silent). Something about them, maybe related to said propulsive field, has a side effect of inhibiting neural activity in the vicinity, with the effect that any animal (i.e. humans, dragons, etc.) nearby either falls asleep, or is unable to wake up if already asleep.
These were originally created by unknown beings to rapidly build/reproduce, relocate, and repair usable structures... but they seem to have developed some sort of glitch, because there seems to be no rhyme or reason behind their building and relocating things, and the things they build and/or relocate don't seem to be in the condition the creators of the mechs probably intended. (Alternate explanation: they are mindlessly copying/relocating things they find in their environment, which happen to be "ruins". Who knows what they were originally supposed to build?)
Depending on what "flavor" you're aiming for, you could substitute "repair mechs" with something like "dwarves" or "fairies".
Alternatively, you could use nanomachines only as disassemblers, with the same "disrupts nervous systems" idea explaining why they're rarely or never seen in action. Maybe ruins don't "relocate"; rather, the nanomachines have some number of templates, and what people think is relocation is just the same template being constructed at a different location.
I think this not only works well for your purposes (no obvious mechanism for the ruins to move on their own, nor are they made of "unusual" materials like nanomachines), and also has tremendous potential for additional narrative. Where did the mechs come from? Why are they behaving the way they are? What happens if one malfunctions and someone is able to "catch" it? You don't have to explore these ideas, but you could.
OTOH... maybe your world just operates as in your question, and no one knows why. I think that could be interesting too. (Even if you as an author have a reason. Of course, as the author, knowing why probably helps for narrative consistency.)
On a related note, I think you're going to have problems with no one ever seeing this happen. As soon as your locals invent any sort of telescope, some one is going to be determined enough to systematically stake out some ruins to see what happens. The only way to avoid this is if whatever causes them to move has some form of near-omniscient knowledge of whether it's being observed.
HOWEVER... this isn't necessarily a bad thing, and could be story fodder all on its own. Given the above suggestions, imagine if pieces of the ruin start to vanish into thin air, or if the whole thing turns to dust.