Smart Bullets
The best thing you can shoot out of a gun is not made out of any particular single material. That is scarcity thinking. The best conceivable bullet will be something more like an M982 Excalibur shell... but smaller. The high tech equipment on these shells makes them cost the US military about a quarter million dollars per shot as opposed to the M107 dumb fire version which costs a mere 330\$ per shot... and yet, even at nearly 1000x the cost of a normal shell, they are worth it to anyone who can afford them because that is how big the advantage is of a weapon that kills on the 1st shot vs the 20th.
The reason you want to focus on smart bullets is that, the question of if and where you can hit someone is much more important than how hard. Imagine an enemy combatant standing 900m down range from you. He's covered head to toe in heavy body armor. You have a choice too make:
Option A is an anti-armor round that can pierce his armor not matter where it hit, but at 900m you'd have to be a world class marksman to hit him at all.
Option B has less penetration, but once it leaves your barrel, it identifies a narrow opening where the enemy's visor is. As it flies it guides its way avoiding all that heavy armor that could have stopped it to punch through the thin plastic visor scoring a perfect headshot even though your aim was off by 10m.
Obviously the smart bullet wins even though the materials you need to make it are not necessarily what you would normally consider great bullet making alloys.
But what about the actual penetrator part?
At the tip of the bullet has to be something that does the actual killing, so we can't just automatically dismiss what this material may be. It has to be something, but it's import to remember what is ideal for a dumb bullet is not necessarily going to be what is ideal for a smart bullet. With a dumb bullet you want a lot of speed and inertia, but in a smart bullet, speed increases your turning arc, and inertia makes turning require more energy. With a dumb bullet you need something soft enough to engage the riffling, but with a smart bullet, you are better off with a smoothbore to reduce heat and barrel ware; so, you are less restricted by such things.
So, keeping all these factors in mind, we know that answers citing gold are based on factors that don't apply to a smart bullet, and answers citing Steel, Tungsten, and DU penetrators are all based on a reliance on a high velocity impact. Now a smart bullet that can correct 10m over the course of a 900m shot would be a great application for one of these 3 penetrator materials, but what if you want a REALLY smart bullet: like one that you can shoot though an open window, then turn around inside a small room to hit the guy hiding behind a wall. Such a bullet would be amazing, but could not possibly hope to move at supersonic speeds on impact after such a maneuver.
Instead, I would suggest a high explosive armor penetrating tip similar to a miniaturized HEAT or HESH shell for your shorter range, higher maneuverability smart bullets. While a kinetic penetrator needs to hit at high speeds, bullets that rely on high explosives can kill even on a low speed impact following a tight guided maneuver.