The wormholes themselves probably don't violate causality, and are mildly plausible. However, they cheat by changing the topology of spacetime itself such that no objects technically ever travel faster than light.
But more generally, you absolutely have to obey that limitation. Not even information may travel faster than light (unless you scribble it down and send it through the wormhole). This makes your premise suspect without either just abandoning the pretense of hard science fiction or appealing to some higher level of physics still.
That might not be the worst approach. I mean, after all, do the users even "speak" the way we do? We vibrate folds of meat in our throats which in turn cause sound waves in atmospheres. What kind of user interface is that? Does it care about pitch? If my timing's off by 110ms on speaking these syllables (because I'm lucky enough that I can speak them), does it fail? How close to perfect do the commands have to be?
Does this require that diatomic oxygen molecules have to resonate with these sound waves? What if it's a helium atmosphere? Or even that it's some funky isotope of oxygen? What lag does this interface have?
Well, unless you want to explore that stuff (and it soon becomes frustrating unless you're Greg Egan or something), then the simplest way to deal with all this stuff is to say that it does not matter. The correct patterns in any medium and at any speed will initiate the magic. You and I would speak the invocations, but the bug people of Zebulon VI would light up their carapaces in blue pulsing lights and either is good.
Thus, it's less of a listening device, and more of a "monitor". The universe seems to proceed in a manner reminiscent of computing. Whatever goes on at the level below that which we can perceive or even imagine seems to run a sort of software that calculates where this electron should be, or how much energy that photon has. Is it time for the neutrino to turn into a tau neutrino? Well, in such a case, the listener might well be just another aspect of the software that does that.
Does this mean the universe is a simulation? Not necessarily, or at least not the sort you're familiar with from popular fiction. That always posits a "real" reality, where ours is virtual. But the scenario I illustrate is more of one where there is no such real reality, and this "software" just executes at the bottom level of physics. Little perturbations in string theory strings, maybe.
Does this mean that some aliens or gods created the universe? Again, not necessarily. It might be possible with some extraordinarily advanced science to alter that software. In which case they did so billions of years ago before anyone else figured it out. Or, if you prefer, no one did it. And the universe just manifested that way without someone orchestrating it (though, from what I read of Wolfram, he contends that such universes are unlikely to develop with any sort of deep magic... if you're appealing to him, you might be down one book sale).
What limitations does this impose on your scheme? Very few. If it's not a simulation, then it's more of a state machine and not a full-blown-computer. In which case the invocation will have to be somewhat constrained. The gas balloon people of New Jupiter who live 100 million years and take 10,000 years to say hello... they might not be able to speak quickly enough for it to work. Nor could the pattern itself be spread across megalightyears and work, for the Gigantor race of the Xeegene Supercluster who scribble with stars the way you and I play with Lego blocks.
The real question is whether your characters cheat and figure out to harvest usable energy from the scheme or not. Sure, a gradeschooler would figure out to just open a gateway to the interior of a large star and siphon off a few stellar masses per day (kindergarteners in your story are probably Kardashev II before they graduate to 1st grade), but if the CGI thingy is creating photons ex nihilo, then you could really do some truly interesting things.
