The Answer Depends on Why You're Asking
Whether for role-playing or fiction, a world building exercise must remember that plausibility alone means little. Just because a thing has an explanation, won’t necessarily make it satisfying to an audience (or players). Science and logic may have their rules, but storytelling has rules of its own. For example, storytelling hates irrelevancy and unnecessary details.
Your question seeks a way to make metals (things that players tend to find desirable, even necessary) available, yet somehow unavailable. This seems a role-playing equivalent of violating Chekhov’s gun.
Chekhov’s gun is the narrative principle that if a scene contains a
loaded gun hanging on a wall, then the gun must go off later in the
story. Otherwise why mention it?
What we’re talking about here is irrelevancy. Story’s are not fictional transcripts listing all things and every detail. Only things germane to the plot or characters are mentioned at all. Game worlds work the same way, describing only the things that are different (from the player’s real world) and that would effect game play. Extraneous details are dangerous precisely because players assume they must mean something. Later when the plot proves them irrelevant, players tend to obsess.
So I see three possibilities for an answer to your question based on WHY you seek to have, yet not provide metals.
Possibility 1 - You want to provide metals later.
Initially your players will find a world largely devoid of accessible metals. Those metal items they crave (swords, armor, guns) just aren’t there. But at some later point, you want to introduce them. If this is the case, the mysterious force that’s hiding your metals is DIRT.
Like our own real world, the valuable stuff is hidden below vast amounts of undesirable dirt and rocks. Every enormous cash of metals in our own history, was at some past point unfound. Everybody living in California in the 1800’s knew there was no gold in the Russian river. Then one day, someone found gold - Bam!, gold rush. In a story or game campaign, such an event would be a plot point.
Possibility 2 - You just don’t want metal.
If you don’t want metals then tell the players there aren't any. I could point out that this is perfectly logical. The non-fiction book, “Rare Earth” seeks to address the requirements for life by looking at Earth and our solar system as an example. One of the issues it covered is that the Sun is unusually rich in metals compared to other stars (which is why Earth is unusually rick in metals). Surely there are other systems atypically wanting in metals. No further explanation needed.
But this isn’t my point. The players may ask, but they don’t really care why there are no metals. They just need to know that metals are unavailable. Once they do, they’ll accept it as a reality of your world and move on.
Possibility 3 - You’re over thinking it.
Your goal is simply to create a world without certain things made of metal (swords, armor, guns). If that’s the case, there’s a simple answer - nobody’s invented metallurgy yet - or those particular things yet.
Or maybe you’re focusing too much on elaborate explanations for your world’s daily reality. As you said, “…I didn’t want to do things the “easy way” and have metal be scarce.” Why not? All fiction is a lie and the key to an effective lie is simplicity, not a web of interlinked elaborations begging for followup questions.
Or maybe the problem is thinking the explanations are what’s interesting to players - it isn’t - That’s YOUR interest as world-builder. What interests players are challenges and opportunities for their characters.
The world that player characters inhabit only needs to be understandable. If they smell an opportunity in the form of gold, iron, or copper then they’ll seek it. If you tell them there’s a reason they can’t, they’ll seek a work-around. Once they finally learn nothing will work, they’ll simply ask, “If we can't have the metal, why'd you say it was there?”
And they'd be right to ask. It's an irrelevancy. That's the Chekhov’s gun part.