Magicians are the Other.
http://motlc.wiesenthal.com/site/pp.asp?c=gvKVLcMVIuG&b=394893
Moreover, we do not know to this day which devil has brought them (Jews) here...like a plague, pestilence, pure misfortune in our country." Martin Luther, About the Jews and Their Lies,
Jewish communities existed continuously in Europe for over 2,000 years. Many of these communities were older than the countries in which they existed. Nevertheless, as the countries of Europe developed, Jews were rarely given complete citizenship status. At best they were tolerated as guests. Their social and religious distinctiveness made them persistent targets for persecution; and such persecution, in turn, intensified the cohesiveness of Jewish communities.
Like your magicians, European Jews had a lot to offer and did offer - urban jews were often literate, educated professionals who contributed much to the communities they were in. They were still hated. I once thought that they were hated because they were economically well off but the same Europeans who hated jews also hated gypsies, who were poor and disadvantaged.
Both groups were hated because they were recognizably different, and because the members of the groups embraced those differences. Hating "the Other" reinforces cultural solidarity among the people doing the hating, a principle which unfortunately is still strong in modern societies. Hate can be a unifying force.
So too your magicians. They are recognizably magicians. Despite their power and potential to be helpful, they are hated as "the other".