What would make a substance that powers magic less powerful when it is moved to a dimension where magic goes unacknowledged? I want the source - a mineral or chemical - to become something that is used only for destructive purposes.
Only being used for destructive purposes is kind of tricky; humans tend to be very creative about getting some kind of valuable work out of destructive events (and vice versa). So I'll focus on the other half of the question - making it less powerful away from home.
As Alexander's comment points out this really relies on where the strength comes from in the first place. Anything that is intrinsic to the world it originates from is a good starting point.
For instance, you could posit that large quantities of magically-active minerals in the planet form a kind of standing magical field - much like how the iron in the Earth creates its magnetic field. Your magical material, when suitably refined, would be able to manipulate this standing field in useful ways. No standing field = no magic, or very much weakened magic (as it relies on the magical "potential" of what you can bring with you, which wouldn't be much).
Or you could imagine the magical material as having some kind of biological source: magic works because the material resonates with some similar substance that's found in the body. Therefore, it won't work against a target from another world who has none of the substance in them.
Both of these ideas will get you a sharp-line distinction between "world where magic works" and "world where it doesn't". However, it doesn't tie the lack of magic into a lack of belief. You could reverse that connection of course - "in this world, nobody believes in magic because it doesn't work", rather than "in this world, magic doesn't work because nobody believes in it". If you want to preserve the latter idea, though, ultimately your magic has to come from belief in the first place: the magical material's properties stem from, or at least are amplified by, the fact that the human population by and large believes that it works (because of folklore, or perhaps a story deliberately spread by someone hoping to make their magic more powerful).
In essence, the people are what's magical, but their belief imbues some of that magic into the material. There are a couple ways this could work at the boundary between believers and non-believers: maybe it's simply weaker, or the skeptics' disbelief somehow passively repels the effect (because that's the effect of their innate magic), or what have you.
Any of these systems can provide a strong boundary between "worlds where magic works" and "worlds where it doesn't" (although they act differently at that boundary: geophysical magic will presumably work on non-believer standing on the magic planet, and not work on believers on the non-magic planet, whereas a biology-based magic would be the opposite).