The proposed situation requires three distinct spheres, none fully comprehended – in either sense of the word – by the others, individually or in tandem. For the sake of argument, let’s presume that “science/technology” resembles its correlate in our world. The question, then, is how to construct the other two poles.
Religion
As noted elsewhere on this site, the common, vague set of presuppositions most people in the West ascribe to the category “religion” are extremely problematic. They do not effectively describe most religious phenomena in human history, and they take as normative a set of Christian theological notions, largely arising from the Protestant Reformation. The result is that all “religions” are presumed to be Protestantism in funny hats.
If you want to begin with the assumption that “religion” is a genuine, legitimate phenomenon in the world, and it is not entirely dependent on such Judeo-Christian notions as “faith,” then you have a phenomenon that can be studied scientifically but which cannot be falsified. This is extremely important. To whatever extent “religion” makes falsifiable claims, they must be presumed incidental to what the system provides its adherents. As Émile Durkheim put it, “No human institution can rest on error or falsehood, or it could not endure.”
The core principle, I suggest, is that “religion” is a phenomenon that binds together social groups and causes them to adhere to and obey abstractions rather than individual desires. Ordinarily, these abstractions are projected into the metaphysical or super/supra-natural. Within a fantasy context, it makes little intrinsic difference, since demonstrable supernatural effects can be analyzed from a scientific perspective, drawing them into the realm of the putatively natural (though see below, on magic).
What science cannot do is to demonstrate that the abstract, metaphysical claims of religious adherents are simply true or untrue. They are not reducible in this way.
To take an obvious example: does the American Flag represent the United States and the people who consider themselves Americans? That’s not in itself an answerable question. What do you mean by “represent”? And yet it is precisely without such refinements that the flag gets so hotly contested (flag-burning amendments, etc.). Insofar as the abstraction functions within the socio-religious context, it does so at an immediate, doxic level. Example: are American flag stamps patriotic? Lots of people think so, including the US Postal System, but they do in fact violate several articles of US law regarding flags and their display. Does that matter? In what sense “matter”?
Some will object that this is patriotism, not religion, thereby demonstrating that they’ve missed the point, but let’s allow it for the moment. Consider Lynch v. Donnelly, the “Pawtucket Crèche Case” (see Wikipedia or whatever for references). In essence, the city placed a “holiday display,” including a “holy family crèche,” on public land. The city said that they did this to bring people together in a positive spirit and prompt them to spend money at the mall. The Supreme Court ruled that this display was not religious, because it was really about money, and therefore secular.
But in what sense is money or economics intrinsically secular? If people gather annually and are compelled to use money to designate a complex system of obligations (e.g., “think of the homeless at this time of year, please give”—should we not think of them at other times? Why the obligation only in this season?), and if infinite TV specials and whatnot insist that money is not what’s really at stake in all the monetary exchanges, then isn’t money the underlying material symbol of the holiday spirit in the US?
Returning briefly to the tripartite division in question, we can see here that a scientific study (such as Durkheim’s) of a religious system is perfectly possible, but it in no sense comprehends, much less explains or explains away that system. It offers a translation from one system of knowledge to another.
Magic
As a rule, terms cognate with the Latinate “magic” (magia, etc.) function largely to designate minority, small-scale, or despised ritual phenomena. At the same time, magia commonly designates the study and manipulation of phenomena outside natural-philosophical expectation.
For example, in the high Middle Ages, the study and use of occult (hidden, infra- or supra-natural) powers was divided into natural magic and demonic magic. Natural magic examined real, identifiable phenomena whose mechanisms, though occult, occurred without the intervention of intelligences. For example: magnetism, sympathetic resonance, astrological influence. Demonic magic concerned phenomena whose causes lay within the power of intelligent, non-human beings (demons, devils, elementals, angels, etc.).
From this perspective, focusing on the “natural,” we can further divide phenomena into those whose mechanisms are knowable but as yet unknown, and those whose mechanisms are intrinsically unknowable because exterior to nature. To understand this, you must recall that “nature” in the Middle Ages meant the sublunary sphere, the world beneath the moon. All powers superior to this, from the moon to the stars and on to the empyrean, simply could not be interpreted fully, because they were not subject to natural law.
In many understandings, such phenomena as magnetism manifested celestial (moon through stars) power within the sublunary. As such, their causes could not be codified or understood properly within any scientific system. Their study and manipulation was therefore intrinsically magical.
You may say, however, that this simply divides “science” arbitrarily into two sectors. This is not entirely the case. If we postulate that a range of phenomena are caused by suprahuman intelligences—gods, angels, etc.—then scientific study may not be useful. Each phenomenon of this kind would be unique, dependent on the intelligence in question, responding to the investigator’s own spiritual qualities. One could never reduce the variables and factors sufficiently to develop controlled experiments that could yield worthwhile results. (They’d work, but they’d give only negative results.) So their study, while theoretically possible from a scientific perspective would be irrelevant and pointless.
Religion and Magic
There will likely be some conflict between these, insofar as magical study seeks to manipulate the powers that undergird human reality. At the same time, this conflict is not intrinsic. From a common early modern (late Renaissance) perspective, the serious magician cannot achieve any higher knowledge of celestial intelligences without their approval, and as they are of the angelic orders, that approval is a kind of divine sanction. In other words, magic only proceeds effectively if the magician fully submits to the deepest personal sanctity.
This is not to say that his work will put him in a strong religious position institutionally, only that he need not be a pariah. Similarly, there is no particular reason that priests or whatever should find him especially threatening—nor that they should take much interest in his results, since after all they have nothing to do with how anyone but he can or should live.
By this construction, these systems certainly overlap, but at a practical level they need have little to do with one another.
Conclusions
There are many other possibilities. I sketch this rough western-European system because (a) I know a lot about it, and (b) it’s easy to research. But it should give you some ideas for how to develop your own system.
The crucial point, as I see it, is to recognize that these three spheres do not intrinsically have much to do with one another. So long as you maintain that at a logical level, you can readily invent all kinds of conflicts and twisted alliances made at the human level. In other words, the vast complexities necessarily arising as these many institutions and groups and individuals come together have nothing to do with what must happen, and everything to do with ordinary human politics and so on.