I know that with a vacuum, you'll be able to gain a whole lot more lift than with any gas. I'm just wondering how you'd do it. What would be needed to make a vacuum balloon as opposed to a hot-air balloon?
The other answers discuss normal matter physical materials, where it seems unlikely to be possible or practical.
However, that's not the only option. As mentioned on other posts (shields, super materials, etc.). I have been convinced that practical warm superconductors will be an enabling and transformative technology.
Besides simply saying "build the ball out of the afore-mentioned supermaterial", this case is simple, and similar to the original application described for the technology in space construction: make a loop of superconductive rope and install a permanent current. Its self-magnetism will turn the wire into a hoop.
The awesome thing about magnetic structures is that the strength is unlimited in the classic sense. Anything that would deform the structure applies energy that's used to resist the movement instead! A hoop, used to make a rib of a balloon, will distribute the pressure uniformly and have no weak point to fail at, since it's just magnetism and electric current.
What's really cool is that the physical material is just flexible string. When turned off, it is shipped flat and folded into a compact container. To use, it is charged with electricity. Flux-pinning technology is used mount the balloons and other vehicle components together.
The strength needed for any material to sustain a vacuum in any meaningfull volume while at the same time keeping buoyancy in our atmosphere is way above current material technology knowledge. You need strong walls to hold a vacuum against the atmospheric pressure, at the same time you need light enough walls, so as to avoid adding weight to the system and end up having something that is still heavier than air. Buoyancy is achieved when the volume of the system vacuum + walls weight less than the same volume of air.
This is simply not possible with known materials.
The idea has been around for about 345 years in the form of a Vacuum Airship. As the Wikipedia page goes on to say, even a diamond sphere can not be used to hold a volume of vacuum large enough to displace its own mass in air.
However, this patent, claims it is possible with a honeycomb material for the spherical shell. But if that's true, one wonders why we haven't seen any vacuum airships flying around. But, it's likely due to the possible violent implosion of the spheres.
Perhaps when we run out of helium, this will be further investigated and we'll start getting vacuum dirigible drones delivering Amazon packages (not blimps, as Burki pointed out, as the shell will necessarily be rigid).
I've been considering the possibility of carbon nanospheres. If they are formed in a vacuum, they'd contain vacuum. They'd have to be fairly large (I'm not mathematician enough to calculate how large), but if they were large enough and didn't collapse, they'd be lighter than an equivalent volume of air or even hydrogen. However I'm not sure if they'd be strong enough to withstand air pressure, though carbon compounds are pretty strong.
If carbon nanospheres are strong enough to withstand atmospheric pressure, then you'd basically fill a container with a whole lot of them, and presto! It floats.
The advantage of nanospheres is that if your container is damaged, you might lose some of your nanospheres, but you are unlikely to lose all of them. The nanospheres may even stay in a compromised container if they are large enough and the holes in the container are small enough, so such a system would be damage tolerant, to the degree that you wouldn't have a near-instant loss of lift.