Depending on what you're using the minerals for, this might not only be worth it, but it might be the only possible way to access minerals in any quantity.
Let's start by taking a look at Apollo 11. The Command / Service modules of the Apollo 11 mission came in at nearly 29 metric tonnes and carried a fuel load (combined kerosine and liquid O2) of around 2.4 million litres to get it on its way to the Moon. I'll point out here that a ratio of mass to fuel load isn't necessarily helpful here as Saturn V was designed from the ground up to get that mass and size into orbit, but this ratio does show that even if you were shipping equivalent amounts of mass (one load at a time) into orbit to make something big, it's going to take massive amounts of fuel.
So; let's build something big.
The Constitution class Starship (TOS Enterprise) has several mass estimates that put it in the 180k metric tonne range. This is approximately 6,207 times the mass of the Apollo 11 Command & Service Module launch weight. The cost of launching 17 Apollo missions was considered too much for many Americans and the program was shut down (in part) because of the ongoing costs to the USA. Just getting the raw mass for a starship (not to mention the building crews, habitats, construction frames and gantries, etc.) would be 365 times the cost in fuel of the entire Apollo program. All this assumes that there's this much fuel to spare from our dwindling resources. Not practical, and potentially not possible.
BUT; we have an entire asteroid belt out there. The advantage (as sdfgeoff points out in his comment) of asteroids is that they have very little gravity to contend with. So, if the minerals are out there in the asteroids, then mining and refining them out there is a better option than trying to get all those minerals off the Earth.
The problem is going to be size and mineral content. The Asteroid belt between Mars and Jupiter comes in at a mass of around 4% that of the moon, and we don't know what percentage of that is usable in terms of minerals. Also, much of it is spread out once you get past Ceres and a couple of other really large asteroids making Ceres the obvious starting place. If we're talking around 50% mineral content (and that's likely very generous) you'd have more than enough materials for a fleet of starships, but you have to refine the materials first. That means smelters, factories, and energy. That's the biggest issue as we'd need energy out there and as there is no mass extinction of plants on Ceres recorded, there's no oil or other chemical energy store.
If we find reserves of Helium 3 though, in quantities similar to those found on the moon, you've got a massive amount (for a time at least) of minerals and energy available with very little energy cost in terms of gravity wells. You build your infrastructure first (refineries, factories, construction frames et al) and then whatever materials you have left go into ships or other space borne needs.
Eventually, we'd mine out the asteroids and then we'd have to look for other mineral sources in low gravity wells. One that we might consider would be the Moon, as even though there's an energy cost to getting minerals off it it's nowhere near as expensive as getting them off the Earth. Better yet, we'd look at the smaller moons around Mars or even the outer gas and ice giants, but then we'd need to find Helium 3 or some other mineral based energy reserve as well and those sites are remote at best. If our energy needs are being supplemented initially by solar, then their distance from the sun is also problematic.
Ultimately this problem comes down availability; not just in terms of the mineral resources themselves, but also the energy required to extract and transport them.
Mining Minerals on Earth is possible because we have massive energy reserves. Getting them out of the gravity well however will cost a lot of that energy.
Mining Minerals on asteroids would negate the energy cost of getting those minerals into space, but then there's very little energy (outside of a potential fuel for fusion reactors) to use out there in the first place.
So, if there's no energy source on those asteroids, then the answer is probably no. It's not worth it because the cost of the minerals comes at a cost of shipping energy up to those asteroids constantly. If the materials are being used on Earth, then again the answer may well be no, because the cost of getting the mining infrastructure up there in the first place would fund what you want to build.
But, if you're looking to build things in space and have an energy source available locally to the minerals, then it would certainly be worth considering if you're building anything at scale.