As others already pointed out: The mass transfer between Moon and Earth would be several orders of magnitude too low to be of any significance.
Also, any pollution created on the Moon will stay there. The vacuum-gap between the Moon surface and the Earth is far too large for any nasty chemical, radioactive or biological waste on the moon to have any impact on Earths ecosystem. This might actually be a good argument for moving extremely polluting industries to the Moon. The Moon has no ecosystem you could harm. Also, the lack of ground water, atmosphere and animals means that any waste dumped in some crater will stay there and not endanger anything living in a habitat nearby.
However, an ecological impact which needs to be addressed is that of the space industry on Earth which is required to build and maintain an industry on the moon. Currently we do not launch that many rockets. But when we want to build an economy on the moon, we will have to launch magnitudes more of them.
The pollution directly generated by a rocket launch depends on the kind of rocket. Some rocket engines run on liquid oxygen and liquid hydrogen which burns into water vapor. While water vapor itself is quite harmless, a large amount of it might affect weather patterns. Also, making that fuel requires a large amount of energy, and that energy needs to come from somewhere. Other rockets, like those which supply the International Space Station, run on kerosine, which is an oil-based fossil fuel. The problems with burning oil are well-known. And then there are rockets which run on much more nasty chemicals, like solid rocket boosters or hypergolic fuels, which are toxic, corrosive, carciogenic and what not.
Also, our current expendable rocket systems drop their lower stages into oceans where they usually remain.
And don't forget about the whole industry which builds the rockets. Their environmental impact also needs to be addressed.
You might wonder "but rockets are so 20th century - can't we travel to the moon in a more elegant way?". Well, currently they are still the best thing we have. An alternative would be a space elevator. The cost of building one would be substantial, but when it is finished it could be used to bring huge amounts of payload into geostationary orbit for a fraction of the cost and environmental impact of expendable rockets. But building one requires materials with properties which are still science fiction.