1. The traditional answer: crazy powers are not all
The usual answer - for vampires, werewolves, Illuminati etc. - is that if the normal humans knew of this minority, they would blame them for every weird thing, and/or attempt to enslave and exploit them, and/or engage in racial cleansing -- and numbers will trump power every time: crazy powers are not all.
This is obviously often, but not necessarily, true - for a real life example see the conquistadores in Mesoamerica. They were a powerful minority, and they still defeated the warlike Mexica by exploiting their limits, divisions and beliefs, with the help of their crazy gunpowder powers.
Magic users are uniquely suited to do much worse, and, being still human, I don't think they would be satisfied with "living and letting live".
So the choice would be between living hidden like rats, cowering in fear of discovery, while Muggles grow more and more powerful and technically savvy, or coming out to a war of extinction. Not an occasional castle-blowing, but WW III.
A third option, the best of a very bad lot, is: strike first, strike hard, leave no survivors.
A magically enhanced plague would be my immediate choice - a few small modifications to some of the nastiest variants of variola maior to keep it slowly replicating, infective, yet completely asymptomatic for some months - then become virulent all over the world in the same instant, killing Muggles swiftly through systemic shock (or a brain haemorrhage). This would almost certainly defuse any nuclear threat, leaving a few days for prepared teams of magical engineers to safely shutdown power plants, stop trains, drain dams, pulverize wandering planes while still on autopilot, and prevent other kinds of accidents.
"A minority of powerful human Hogwarts-style wizards" == "Black Death apocalypse" (details may vary). So - why hasn't it happened (yet)?
2. A different answer: crazy powers are crazier than you thought
Especially if they have powers like invisibility or infinite armor.
That might be the point. They might not have - that, or anything else. Magic might turn out to be a weird quantum phenomenon that does not survive skepticism.
The need for a Magic Academy arises from the necessity of teaching young children to believe in magic while they're young, lest their magic disappear - which is what happens usually to Muggles.
At length the Man perceives it die away,
And fade into the light of common day. - - W. Wordsworth
Once the belief has taken roots, the children will grow able to perform magic - provided they are in sight only of other magic believers.
But let one Muggle witness the scene, and think to himself "Hey, what's that weirdo attempting to do? Doesn't he know brooms don't fly?" - and the broom won't fly, the Elven gold will become rotten leaves. That's the reason for the warnings in the old tales about not opening boxes and doors, and not looking at magic being performed. Just like faith moves mountains, so lack of faith makes them collapse.
So the magic users, if they want to reap the benefits of magic (and they do), must perforce live that part of their life in utter secrecy.
Most magicked items and effects will survive Muggle skepticism, provided they have a mundane explanation. Unfortunately, the readiest explanation for large amounts of money and trinkets is grand theft or drug dealing, so while a magic user will be able to live the good life, he must be very careful and is thus not free to use his magic to the fullest, except in private.
This is shamelessly "inspired" from David Brin's Those Eyes!, wherein the magic/alien/elves cannot abide human skepticism.
Three of those young faces still exhibit rapture as they stand uncritical, accepting. But the fourth — a gangling child-woman — casts another kind of glow. As she rouses, her eyes narrow, and her mouth forms words. Tapped into her mind, I sense her effort to see. To really see.
"What am I staring at? Why... it looks transparent, as if it isn't really there at -"
"Flee!" Fyrfalcon screams, as we are blinded by that deadly gaze.
[...]
By now those teenagers are rubbing their eyes, already convinced we
were hallucinations. That is what happens when humans see us with
skepticism. Now we blow away like leaves, like wisps of shredded
dreams.