I'd think about your characters, and perhaps future generations of your characters, and how you will use the world. Do you intend to be using the same world ten years from now, IRL? Then you better make it big; no telling what stories you will be telling in ten years.
To me, the world needs to be logically coherent, but a "setting" is the best kind of deus ex machina. You don't have to explain why a talking coyote guards the cave leading to the elven kingdom, that's just how this world is. Or why your hero has to travel to Septima 7 to consult with a pedantic vegan Tyrannosaur on his warp drive modification. Or why his warp drive didn't do what he wanted in the first place.
In Star Wars, much of the plot and resonance is just about the setting. Luke is in danger in a cave (when with Yoda, he encounters the hallucination or whatever of Darth Vader), while Hans Solo is in danger in a cave (inside the giant worm they thought was a cave). Or, what idiot sets up a military base in Antarctica (Hoth the ice planet)?
Blah blah about strategy, but that was world-building too; making the best strategic place for their base so frikkin' inhospitable. Really Hoth was a deus ex machina, a setting to push Luke to use his Jedi powers.
We can think of settings as villains and helpers. The magic forest will try to kill you. The greenstone fountain will heal you. Ariel's castle is safe haven, if only you can get there, and she knows the secret to the maze you must get through to find the home of the white dragon.
but like characters, you don't want the settings to seem too conveniently helpful or harmful or powerful. When your character is confronted by an evil dragon, you don't have a witch appear, give your character a handful of powder that kills dragons, and then disappear, never to be seen again. Nor can the forest they have entered have some convenient suit of armor discarded in the forest, complete with magical sword, that they can use to defeat the evil dragon.
You are better off describing some far off lands and leaving the world vague (as it is for most of us IRL, anyway). Sketched settings can be like sketched characters; interesting and potentially worthy of their own story in the future, but just introduced in the current story to serve some plot purpose, like an excuse for knowing something critical, or to go somewhere.
Walking on the dirt road, they passed a thin path veering left, marked by a head-sized black rock. Callie shifted the sword on her back and nodded toward it. "That one there winds past the black mountain, a few days down, close on the Bent Village. My mother was born there. Said it was the most horrible place."
Marc looked down the path, empty to the horizon. "A damn small mountain, or more'n a few days. What's horrible about it?"
"She never says it, and too late now. Heard tell they mine crystals there, and the tunnels are like black glass, so you can see your spirit in them. Like on a smooth lake in the moonlight. Always wanted to go there."