Well of course almost any religious believer would say that the God or gods he believes in DO affect the world. So how is what you are suggesting different from real life?
(a) There is no question or ambiguity? The gods come down from Mount Olympus on fiery chariots and throw lightning bolts and their enemies, and there is no way that anyone who is not blind could say that this was not done by the gods? And they do this often enough that pretty much everyone in the world sees it regularly? (As opposed to Judaism, Christianity, and Islam, which say that God usually works in more subtle ways, and that "blatant" miracles are relatively rare so that most people only know about them by reading scripture, lives of saints, etc, not personal experience.)
(b) There are multiple competing gods? Like in Graceo-Roman polytheism. (Modern Hinduism has many gods of course, but I don't know enough about Hinduism to say if the believe the gods compete and fight like the Greek gods.)
I can see how (a) and/or (b) could put a strain on freedom of religion. In the real world, an atheist might be annoyed at Christian prayers, but he presumably is not literally afraid that the Christian prayers will result in God bringing down some judgement on him, because he doesn't believe there is a God to do so. But if the existence of these gods was undeniable, then knowing that my neighbor who worships the cow god is praying for judgement on all those who eat beef, I would not dismiss his prayers as vain appeals to an imaginary god. I'd have a real fear that his god would respond and do me harm.
In the real world, if someone worships a different god than mine, I generally believe that his prayers and rituals are just a waste of time. They don't do me any direct harm. (Barring special cases, like he kidnaps me to offer me as a human sacrifice.) But in your proposed scenario, he very well might do me harm. So in the real world, it makes good rational sense to say, let everyone worship in his own way, and attempt to persuade others that his way is best if he so chooses, and see what works out. But in your scenario, you can't just "live and let live". It's not that they're worshipping a false god, it's that they're worshipping a very real god who is inimical to me.
To the extent that the gods compete, I think freedom of religion would become very difficult to sustain. It would be too impractical. Like, I have no problem being tolerant of people with different skin colors or languages or favorite foods. But I do have a problem with being tolerant of people who steal and kill.
If the gods do not compete, if they're all friendly to each other and tell their followers to respect each other, than all this goes away, and freedom of religion becomes like accepting that some people like pizza and some like tacos. There's no reason why I should care if your tastes are different from mine.