All my ideas are really far fetched fantastical, way stranger than one would be used too, crazy unrealistic with its magic, and I worry about the accessibility of these worlds to an audience. Is anything, no matter how far fetched, fair game as long as its self consistent?
The story should be relevant to us as people, it's setting could be as wild as you want (wilder the better) as long as it's self consistent as you say.
The key in your question in my view is 'self-consistent'. You could create a world out of a single idea and this leads you to unforeseen results. The story then comes out of this setting, and as long as its implications are relevant to us as people, it will not alienate the audience even if it is totally wild.
Vernor Vinge, in his Across Realtime stories, had a simple idea: The ability to create bobble universes that isolates an area into a spherical separate universe where time is frozen until it pops. Using this, he created a whole plethora of implications in his world, where humanity fragments through time as bobbled cities burst millions of years later, and people bobble their way through the galaxy. But coupled with this is the story of humans and their unique qualities coping with this reality.
Or his Deepness in the Sky series, which are based on a simple premise that computers are slower and 'dumber' in the middle of galaxies, but ultra-smart at the rim, and god-like outside. This allowed him to create world of conflict and discovery throughout, coupled with survival stories of humans trying to hold onto their identity within it.
Go forth, and find the implications of your craziness. The best stories come from finding where the boundaries are, what the end points could be, and what our place is between them.
There is a reason why one of the commonly cited pieces of advice to authors is murder your darlings. I know it can feel irritating to feel like you're throwing away work, but while it can be extremely fun to think about interesting worldbuilding, it never sells a story on its own. You should also include less of what you use than you think. As a writer, worldbuilding is homework, and homework exists to make the difficult look easy.
A really nice analogy I'm stealing from Mythcreants is that you can think of your story as having a budget of a stack of 36 pennies(similar to the point about economy of innovation). This has to be divided into focusing on plot, characters, theme, and explaining your worldbuilding. So a complicated mystery works better in a modern setting with a relatively uncomplicated detective because it leaves more time dedicated to the mystery itself.
Here is where your unusual premise becomes a problem. The more unusual it is, and the more different ideas you explore, the more of these pennies you'll have to spend explaining it. This means you'll have less to spend on things like character, plot, and theme. Novelty of an unusual premise helps a bit to sell it, but rarely enough to make up for this loss.
People, especially general audiences, don't read books or watch movies for cool worldbuilding alone. They enjoy stories because of the characters in them and the journeys those characters go on. As much as I love cool worldbuilding and exploration of ideas in works like Ghost in the Shell, The Expanse, or The Legend of Korra, if I didn't enjoy the adventures of Motoko, Amos, or Korra none of that would matter.
For every published story there are probably a dozen authors with the same story who spent 20 years worldbuilding instead of actually writing but we’ll never get to hear their names because they’re still too busy fantasising over maps and characters backstories (I love drawing maps!!!).
Be brave, bite the bullet, start writing. Only then will you know.
Ground your weird with the familiar.
@JustSnilloc - has a good example with Wonderland; it is trippy and surreal, but the reader is grounded by Alice and her ordinary perceptions and reactions. Riddley Walker is written in a very difficult future patois but the world it describes is not that bizarre. Kafka's Metamorphosis has one super weird element, and the rest of the world is built around and shaped by that weirdness. A very strange musical piece with dissonant and jarring chords and tones can be grounded by a familiar time signature and beat.
Give your reader or player a place to stand that is familiar, from which to survey and appreciate your high weirdness. Even the familiar can be unsettling and different when illuminated by the weird and novel.
Decide who you're writing for.
There is an old saying: write the kinds of stories you like to read. If you like far-fetched and fantastical, the strange beyond the wildest imagination, that that is what you should write and that is how you should worldbuild. A world does not have to be consistent within itself, though whether or no, it should be well made and enticing for others to enter and discover what is to be found.
This is called art.
There is another old saying: I like to eat. If writing is your primary occupation and your primary source of income, then chances are you will be working with editors and publishers and will have in stead to "write to the audience".
This is called business.
If you're in this forum, chances are good you worldbuild and write for fun. My advice would be nothing more than to make the worlds you like to imagine and write the stories you like to read!
Your audience will fine you!