I can think of various answers, but first it's necessary to discuss the nature of magic.
The current most common view of magic among fantasy readers (and writers) sees magic as just another talent, rather like the concept of psi powers in the 60's. In this view, magic conforms to at least the outline of conservation laws as expressed in the physical sciences. As Terry Pratchett put it (approximately), "Wizards have to be careful about using magic. If a wizard tries to levitate too large a body, he may drive his brains down into his boots." So, for instance, a good deal of discussion is given to the accumulation of energy and the energy needed to produce various effects. Your own phrase "all sorts of manipulating elements, pure energy" suggests that your own ideas run in this vein.
This is wildly different from the more traditional views of magic. The oldest paradigm, as far as I can tell, is the calling up of demons, of djinns, or whatever you call them. These supernatural beings are of various degrees of puissance, and once you master them by using the proper ritual, their power is at your disposal. So a djinn can build a palace in a night, or provide you with an enormous treasure of gold and jewels. Calling up such beings has its risks, of course.
A somewhat later conception of magic has the rituals themselves as the source of power, for reasons that are never well-defined. Speaking the proper spell in the proper manner produces the desired results. A modern example is the Harry Potter series, in which ritual is critical, but it must also be accompanied by magical talent. This approach also has the peculiarity that everybody's magic is somehow the same, since the rituals (including potions, for instance) work the same for everybody. It's perfectly possible to posit that each individual must develop a ritual vocabulary unique to herself. This would make the development of each magician an idiosyncratic process, and would eliminate the concept of schools of magic. One consequence would be that the development of a strong innate talent would be extraordinarily hazardous.
With these older versions in mind, a few possibilities present themselves.
1) As H.P. Lovecraft put it, "Do not call up that which you cannot put down." Summoned beings are often rather disgruntled at being put to work, and eventually the Atlanteans called one up which they could not control. It was not happy about being interrupted at its normal routine.
2) Magical beings have, shall we say, a robust sense of humor, and like nothing more than to fulfill the letter of a command while disregarding the spirit. There are any number of short stories along this line, dealing with contracts which blow up in the signer's face. Some bright boy in Atlantis wasn't quite as bright as he thought he was.
3) Ritual must be precise. The greater the results the more precision is required. Somebody in Atlantis got over-ambitious, and flubbed a really major spell. He wanted, for instance, sunny warm dry weather across all Atlantis for the King's birthday. He dropped a syllable and got dark and cold and wet, as the surface of Atlantis dropped and the sea rushed in.
4) Magic, which violates the natural order of the physical world (which is why it is called supernatural, or above natural), is subject to a different sort of conservation law. It draws order from chaos; but chaos, like entropy, can only be locally decreased. Continued magic use produces a local condition which is increasingly unstable, and the Atlanteans were eventually overwhelmed when the chaos which they had excluded came crashing down upon them, and the larger balance was restored.
5) There exists among the supernatural community a sort of ecosystem, and the ecosystem (like all ecosystems) contains some remorseless predators. These predators ordinarily ignore our mundane existence - it has neither flavor nor nutritional value. Acts of magic, however, are rather like throwing blood in the ocean when there are sharks in the area. Eventually, Atlantis attracted one of the supernatural world's Great White sharks. Or, if you prefer, a school of piranhas. If it really were a Great White there would have been nothing left of us, so we got lucky that we only attracted the piranhas.