I've read this question, but most of its answers don't seem to apply to my question directly other than possibly this one, which suggests that in order to change a planet's orbit, you would have to change its rotational energy.
There's also this question, but the answers are more cataclysmic than anything.
What I'm imagining is the entire current population of earth (roughly 7.5 billion) being moved to the moon, with new buildings and roads to house and transport every individual both above and as far below the surface as possible. The moving of every person to the moon would be a steady and smooth stream until everyone has been transferred (let's say 100M/week).
So in the event of mass-colonization of our moon, at what point, if any, would it start moving out of its usual orbit? Would it fall or raise? I know there's no drag in space, but would the added height/weight/energy from the billions of bustling people cause its rotation to slow down/speed up at all? Assume all warning signs are ignored and people keep piling on.