There are plenty of modern or slightly futuristic developments of the firearm that would avoid punching a hole in spaceships. The problem is that in doing so, all of them are much worse against body armor.
How could you design ammo that would punch through an enemy's body armor but not punch through the hull of a ship or space station?
Range-detonated munitions.
The US military actually performed field testing of their "XM25 CDTE" which is an airburst grenade launcher with pre-programmable detonation distance. It was designed to target enemies that are hiding behind cover like walls or windows without resorting to the inexact and often infeasible ballistic arcs required by traditional grenade launchers, particularly in urban warfare settings. This is how it works:
Ultimately, the XM25 program was canceled for a variety of reasons but the technology would be perfectly plausible for a near-future setting. Your space firearms could operate something like this:
This avoids overpenetration because even if the bullet hits an unarmored target, it still detonates all the same. Also, if the user were shooting down a long corridor, they could set their gun to disarm all projectiles that fly past the enemy or past a certain range leaving only inert rounds to strike any pressure vessel bulkheads. Unfortunately, even with such a system it would still be possible to purposefully shoot out walls, but I don't think there's anything you can do to prevent that.
In general, I'm struggling to think of a "realistic" space combat scenario where there are boarding actions and people have access to body armor and armor-piercing rounds but they don't have armored walls or they care about maintaining pressure. "Realistically" all the combatants would be wearing armored pressure suits anyways and loosing pressure in the ship/station would just be the price of close quarter combat.
Assuming you need penetration and can’t rely on blunt force trauma.
Spikes
More specifically: flèchettes that carry a payload of sealant at the rear.
If the sealant is expanding and can be separated from the flechette just after impact then you have around that will happily penetrate armour and deliver a metal spike right into it’s squishy target. The same round will also go through the skin of a ship, but will instantly seal the hole it makes while the flechette sails into the void.
If you make the flechette heavier, longer and sharper you can penetrate heavier armour while still sealing the hole you leave.
Basically I’m proposing high tech space crossbows. Go figure.
Explosive bullets.
No, seriously!
A spaceship hull will always be a multilayered construct, as a large part of its job is to keep those Meteor(tm)-brand hypervelocity bullets out there on the outside.
An explosive bullet that chews a neat 1-inch spherical hole in the inner layer of your hull will be an annoyance for the maintenance and repair crew.
The same 1-inch spherical hole in the enemy's body armor will seriously inconvenience him.
Note that the ammunition to be used relies on its explosive power, not on projectile velocity to do its damage. Also, so they can be used in both gravity and zero-g with equal success, you would want a self-propelled bullet, that fires with virtually zero recoil. Something like the Gyrojet.
Conceivably, you could even make the projectile smart enough to only detonate on hitting an appropriate target. Although I have no idea how to fit enough brains into a small projectile like that, there are military minds that can think around corners like that. With projectiles like Smart Bullets, that somewhat aim themselves to a laser-lit target. If you can make them smart enough to aim, you can make them smart enough to not explode if the impact target is invalid.
Unfortunately, there's not much that can be done to prevent a hull from being penetrated by a round that can also penetrate body armor.
Body armor is specifically designed to catch a bullet and slow it down over a distance, which is why the shot still leaves a bruise or broken bones (in some cases).
Even having a steel hull that's 3/4" thick steel isn't a guarantee that it's bullet proof. The .50 cal rounds in the video below aren't armor piercing, yet they very nearly penetrate the steel plate. That's a gigantic amount of mass to have in a spaceship and would take a massive amount of fuel to propel it, as well as try to get it to change direction for "evasive maneuvers". You need Star Trek level tech to make that happen, essentially greatly reducing the momentum caused by the mass and Newton's laws of motion.
https://www.youtube.com/watch?v=9dXvetFDX3A
And a .50 cal handgun can easily penetrate multiple thinner steel car doors, with the glass and the interior still installed, but it's caught by a Kevlar vest. As this is done by a civilian, it's also not an armor piercing round, which is proven by it not going through the vest.
https://www.youtube.com/watch?v=SKpqfXnvFHk
In the US Army Basic Training I had 23 years ago, we were shown the capabilities of a M16A2 armor piercing round before we were given the rifles to practice with. The relevant example here is that a steel ammo box was filled with water and shot. Neither the 2 sides of the box or the water significantly slowed down the 5.56mm round. For comparison, 5.56mm is a similar caliber to a .22 rifle, just with a much higher muzzle velocity and different shape. I wrote more about that here.
And yes, water can have an effect on bullets, just not the amount in an ammo can. And the higher the velocity the faster the bullet fragments in water, however this wasn't tested with armor piercing bullets.
https://mythresults.com/episode34
And if you want to see what a .50 cal armor piercing (AP) round will do, here's another video. Apparently this guy isn't in the US. This is still considered an easily carried weapon for the battlefield, so could easily be used on a ship during boarding. It's also the same caliber as the .50 Desert Eagle of the handgun in a video above, so well within the realm of battle use, but it won't have the same stopping power due to the shorter barrel.
https://www.youtube.com/watch?v=QsgwcDOGJQY
If you think changing the medium of the gun will make a difference, steel is more dense than body armor, so it wouldn't as easily be vaporized with a laser or plasma gun, but then you have surface finishes to deal with.
I wrote another Answer about lasers from my experience with laser cutters in non-professional manufacturing, and even included a video I made testing my theories about lasers and anti-laser armor.
Some of that Answer would also apply to plasma, but you're still dealing with something that is considerably hotter than the melting temp of most materials, so would easily penetrate steel just like body armor. The reason why a plasma cutter or laser cutter works on steel is because it is so much hotter than the melting point that it instantly vaporizes the material in front of it, rather than heating the whole piece.
You might be able to get away with plasma or lasers, due to the body armor being relatively thin, but you'd still need thick hulls, which would still have literally tons of mass. If you go with aluminum, which is more reflective and lighter than steel and would help prevent laser holes, it wouldn't work against plasma or slugs.
If you upgrade your hull with force fields, you might be onto something, but that would take a considerable amount of power, which might be better used to get away from your attacker. And that would probably have prevented your ship from an enemy breach, or at least boarding, anyway.
Also, if you have that level tech to be able to have it everywhere on the inside of your ship, you might be using it for body armor as well. There's this long standing race between armor and things to penetrate armor, with the piercing elements always getting the upper hand faster than the armor can change to protect it's wearer.
https://en.wikipedia.org/wiki/Body_armor
I can't find any article that specifically says it ATM, but in the race between protecting the wearer and killing an enemy, killing the enemy always wins in the end. Nothing is bullet proof, only bullet resistant. Your force field is turned down low to preserve power, so it'll take a 9mm, but not a .45 cal. Or it's turned up to deal with a regular .50 cal, but an AP round goes through. Or it's turned up all the way, and now you can't return fire, because cycling between protection and firing takes too long. Not to mention that dropping your shield to fire your own weapon also allows projectiles in.
There really isn't a practical way to prevent hull breaches from weapons that can pierce body armor. Unfortunately, there's just too many variables involved as well as too many forces to deal with to handle with anything, but magic or handwavium.
An anti-tank rounds can kill a kevlar-wearing person much more easily than regular .45 rounds[citation needed]. Yet you would not you take a 17-pdr to a gun fight, unless you are either Deapool or Rambo. You take the f...... handgun, which would not tickle a ship's hull.
It's all about the right tool for the right job. Even the most Rule of Cool dependent media such as Star Wars honors this, because it is such a given that picturing otherwise would be weird.
Design the ammmo in such a way that it delivers just enough energy to bring down an armored human target but not a polar bear, for example, and spaceships are safe by extension.
You can see the energy output of many different weapons in wikipedia. A hunting rifle for large game might deliver 3,000 joules at point blank, whereas a .357 Magnum delivers 500 to 800 joules depending on the kind of ammo. You might need much more than that to pierce a hole on a ship.
It punches thru either one. With a very tiny hole.
Your gun shoots very small very hard aerodynamic projectiles very fast. Maybe it is a railgun and it shoots osmium needles. These things do go right thru body armor. When they hit a body they produce a shock wave that turns things to jelly. Projectiles continue on out the far side, followed by some jelly in slow motion if it is that kind of fiction.
They also go thru the ships hull and leave a neat 2 mm hole. You can fire at something outside right thru the hull if you choose. Atmosphere can escape thru this hole. It makes a whistling noise. Not a whole lot of atmosphere escapes. Some jelly floating in zero G might get sucked out with a slurping noise, if it is that kind of fiction.
A piece of chewing gum can plug that hole, for the time being or longer as events warrant.
Maybe instead of doing that just have the people getting invaded vent the ships oxygen and other stuff and just wear oxygen supplies so that 1 invaders can just kill them all by wearing oxygen supplies and blowing wholes in the hull and 2 so that they only need to worry about hitting the ships vital parts which maybe they can see with augmented reality so it doesn’t matter if the bullets can penetrate the walls or not. Also if the ship is carrying such little oxygen that venting it will dangerously lower supplies than the crew is already in big trouble.
Keep it Simple
The only practical way without over engineering everything to a ridiculous degree or bringing 'unobtainium' into the mix is to make the walls of your station (and by walls I means the external hull, viewing ports & airlocks etc) from material that is harder or thicker than anything which can be practically be warn by a human being.
So to use a modern comparison; the maximum current rating for body armor in the US are Type 3 & 4 condition body armors. These are basically plates that will stop at least several high velocity rifle rounds before failing.) But to the best of my knowledge no-one makes conventional combat armor that provides full 360 degree level 4 protection. This is because it would simply be too heavy and cumbersome to wear for any length of time. Instead the highest levels of protection are limited to the vital organs and everything else is usually made of more flexible materials that offer lower levels of protection.
Even the current US military body armor, which does not offer that same level of all round protection has caused problems with something like 30% of all injuries in the field (don't quote me on that stat) being put down to strain caused by the weight of the equipment and armor soldiers are required to wear for long periods of time.
So make your exterior of the station out materials that cumulatively give you levels of protection comparable to '4 plus' and your good to go. And it doesn't have to be steel etc. For example a composite wall consisting of a thin outer metal hull lined internally with a layer of carbon fiber armor, then perhaps an thick layer of aerogel and finishing with a thin inner wall might do the trick.
