...But Religion!
"Because Religion" is still a valid reason, it's just not a single blanket reason that might span all religions.
The gods might not care about how mortals use magic, but the mortals administrating their deity's will do. And since the gods are morally ambiguous on the matter, it is up to the mortals that interpret their deity's will based on other aspects or items that their deity does (appear to) care about in order to give context to the matter in the absence of their deity literally coming down to lay down Divine Law.
Thus, the mythologies and tales surrounding the religion paint it as an evil act, because it violates some other core tenet of the religion. To put it another way, casting this way goes against a deity's portfolio/philosophy, or at least the human's interpretation of it.
Casting an impromptu spell in haste may be heretical to a deity whose core tenets include preparation and diligence. The tales will frame it as evil due to not following that tenet of the deity's teachings than a failure in the magics.
Likewise, a deity that rewards sacrifice might see the lack of material components as anathema and might have a mythology of punishing mortals who sought magical might without apparent sacrifice.
Do NOT Try This at Home!
An alterative is that as magic developed, the components and circles and other paraphernalia created enough buffers and safety valves for spellcasting that failure did not outright kill anyone, though bad things still did happen. Then some aspiring archmage a long time ago decided to try doing his magic without these safety components.
Saying that "It did not end well" is a understatement. A large one … almost as large as the crater they left behind. Nobody ever did know what spell failed (or succeeded) to get that crater. All they know was that it was massive, there was great loss of life, and the next year or two suffered lower harvests.
Ever since that day apprentices were taught that casting without components is a Thing You Do Not Do. Unless learning the magical arts are centralized in one place, each apprentice may be told a different yet equally gruesome reason why you never omit components for casting.
Even if someone figures out the secrets, they will be treated as a ticking time bomb -- a person that will imminently turn themselves and a large area around them into a crater. They will be shunned, and possibly even hunted and put down in order to prevent what everyone believes will be the inevitable result -- catastrophe.
Control
Overall, the grand reason why casting without components is "evil" is to control magic through controlling the substances needed to perform it and to help identify possibilities by what components are being used to cast it. It is a societal construct that ensures the status quo remains for as long as possible.
Yes, this does mean that the elite and those that think themselves above this system would try to exploit this, but that itself a balancing act for the elites. If they get caught doing evil magic, they will be branded as irredeemably evil with all the fallout that will entail. It will almost assuredly end bad for them outside of applying overwhelming corrective force to it -- be it physical, social, or magical.
It also entirely means that at some point there might be a magical revolution, as this evil way is discovered and analyzed enough to be not an immediate physical problem. It is also plausible that the world has already had one of these and there were delayed consequences that altered the world as it is/was known, such as this being where the monsters/demons/angels of the world come from.
And above, the gods are uncaring -- so long as their interests in the material realms are being advanced. If they're even paying attention to the world this century.
The world's past might offer insight as to why this is the case