Fear is a powerful motivator.
Magic is a powerful tool, for good, for evil, or for neutral effect.
Grant someone, anyone, the ability to do massive harm to people or property, with little or no promise of protection against that action, and you breed fear.
That's the entire concept behind the Cold War's Mutually Assured Destruction. The idea that someone can rain down fire on your head is something not to be taken lightly.
Or, to take it down a few scale factors... Have you ever seen guards in a public setting holding rifles? Here in the US, that's not a frequent occurrence. But I have seen it. The first time I took a commercial flight after 9/11, National Guard soldiers patrolled the airports with rifles. It was unsettling to see, even though they were the good guys at the airport. I've been in foreign countries where police did the same. It was, again, unsettling.
Now take those rifles and make them invisible. So anyone, anywhere, at any time, could be packing lethal firepower.
That is more than enough to instill fear in people.
Quite honestly, if you don't have magic powers of your own, fear is the only sane way to view powerful magic.
Of course, as Yoda teaches us, "Fear leads to anger, anger leads to hate, hate leads to suffering." That cycle plays into it as well. We hate the things we fear. And we do what we can to make those we hate/fear suffer.