Comprehensive Guide for dealing with boarders
I encountered this issue building my own setting a while back. The tech baseline for this answer is relatively hard sifi, I'm gonna mention alternatives for softer settings though. The ship I'll discuss has no magic gravity system, and is designed for linear thrust and in some sections spin "pseudo gravity". Power is provided by a fusion and backup fission generators, antimatter is availiable in decent quantities. The main propulsion system is a torch drive, capable of acceleration up to 30 G. Shipboard manufacturing is advanced, not on a Star Treck replicator level (even though that would make any tactic involving manufacturing systems easyer), but given the raw material, a blueprint and a thermodynamically reasonable ammount of time there's little that can't be produce. AI is very advanced and handels lots of things on the ship. Nanotechnologie is advanced too, biologically and mechanically. You need to consider if a given tactic fits with your settings technologie.
This category includes anything involving atmospheric manipulation. These methods only work if the boarders where stupid enougth to leave space suits behind.
This is pretty self explanatory.
If you expect trouble beeing able to introduce gassed to pacify people on board is reasonable. There is a wide range from nice to nasty. THC. If the fighting force is stoned there will be little fighting. Knockout gas. Self explanatory. Chlorine or Fluorine gasses. Those are the nastiest available, they should be bad for any kind of life form. They might damage the ship slightly or could be illegal in many systems. Could be explained as part of a decontamination system. There are numerous other options CO, CO2, safrin, zyclon b,...
Again based on human biologie , yet any species should have its limits here. This is the poisonous gas version for civilian vessels. Removing oxygen from the atmosphere is a sure bet. Stopping CO2 removal will be slow, but effective. Overpressure is another way to do this. O2 turns lethal at 1 bar partial pressure, nitrogen at 4 bar and CO2 at 0,005 bar. Of cause overpressure itself is not pleasant. On that note, spacing, rapid depressurisation, and beeing sucked into space is a grat way to kill someone too. To make this more effective just overpressurise the ship. At 20 bar pressure spacing is 20 times more fun! Needless to say this works against spacesuit users, but doesn't kill them.
evil uncle Newton
This category is based arround killing and harassing with velocity and acceleration. Everything here depends on how and how fast your ship can accelerate, how people brace themselves against acceleration (crash couches, inertial dampeners, nanotechnology and or augmentation), how gravity can be produced in the setting (only pseudo gravity or magic gravity) and how much ship and boarding equippment design seeks to counter the dangers from these approaches.
Accelerate as hard as the ship can, spin the spin-sections as hard as possible, turn the gravity magic up or even repulsive, while the crew is savely braced. Everything depends on the limits of these technologies. And on inertial dampeners. Turn them off. Better yet, don't use them in your universe. They are a stupid technology that removes tension from stories. Anyway your boarders ahould be well done after 10 min of 30 G. Or better 10 min of irregular, short 30 G thrust spikes followed by swift counter acceleration. That will bounce them like a flummy between flour and ceiling. You might have to repaint after that though.
- spin the ship up arround an axis it wasn't supposed to spin arround
This was already mentioned in the comments. It won't kill tbe boarders, but it will buy time.
This category is about pre installed weapon systems. It might not be fitting for civilian vessels.
- pre installed anti boarding
Have autonomous or steerable weapons in flour, ceiling and walls. These could include but are not limited to machine guns, low yield railguns, lasers, mines, claymores, blades, spring loaded spikes, flame throwers, acid launcers and rocket launchers. They will obviously damage the ship, but as you know their yields you can disallow them to fire at certain vectors, where they would take out vital systems. So damage can be held managable.
Have a software enabling the override of standard security measures. This software allows you to rapidly open and close doors or airlocks for spacing and crushing boarders. Makes blowing up pipes possible. Have elevators fly out of the ship.
- turn main weapons on the ship
Some of the low yield ship to ship weapons could be turned inwards. Look out that you don't destroy vital systems and this can be very lethal. Railguns or gettlings seem best suited. No matter how good the alien armor is, a slug moving at 5% the speed of light will pierce it.
Heat management is incredibly important on space vessels and criminally underused in sifi.
Different cooling fluids will be used to cool different parts of the vessel. Drowning boarders by flooding overtaken sections might be possible. Freezing them with liquid helium (needed to cool superconductors and stealth hulls) is quite mean. Boiling them with the liquid metal or salt from the drive and reactor cooling system has it's benefits as well.
Remember how I mentioned production capabilites? All sorts of mean stuff can come from there. The anti bording weapons I mentioned could be quickly produced and placed for example.
This depends on the nanotechnology of your setting. A flood of all consuming quicksilver might be on the table. Or invisible flyers, carring millions if micro warheads into the boardes lungs or collecting on their armor for a major strike. Just be careful to get these back under controll.
How about a virus ramping up paranoia and aggression in the borders? One that allows you to take over their motoric functios? Or one that just kills them? Nah, that would be boring.
- hunter kill drones. lots of them
Start producing robots. And don't stop. Your aliens sound like formidable fighters. But quality can be beaten by quantity. If a great knight gets surrounded by 30 peasants with pitchforks, he is dead.
- sacrifice boarded sections
This works better if it is done early. Desinfect the wound before the infection spreads. Overload all system in the area or shot one of your own missiles at the breaching pod and breached section.
This means loosing the ship, but I'll still mention it. Fleeing in a shuttle is the most obvious option. If cryogenics are a thing, decapitate the crew, freeze the heads and put them on a high burn missile on a course back to you allies. If minduploading is a thing you can evacuate via transmission or play ghost in the machine until a chace for retribution arises. If your aliens are all so bloodthirsty that they all join the boarding operation, counter board their ship. Set up hinderances onboard the old one and attempt a full force counterattack. The aliens might be cought off guard, considering their self image. Since you are technologically superior the enemy's computers will surrender quicker to your hacking than your computers will. Then blow up your own ship.
- set loose something worse
In the book Revelation Space Ilia Volyova sets free captain Brannigan, who is infected by a super-intrusive and aggressive nanotechnology virus, the melding plague, to regain controll of her ship from the alien software entity Sunstealer. This of cause requires you to have some sort of lovecraftian horror up your sleeve. And then you need to deal with said horror. Watch out that the cure doesn't become worse than the disease.
If you got augmented reality or some form of none invasive memory manipulation, play withbthe boarders to have them kill themselves.
Implant the crew with antimatter bombs. Then call the aliens and explain the situation. If any of them are injured we all will die together. Can even be held back if the aliens take hostages to threaten to blow up their ship. Can be done with the main reactor too. But will the xenos belive the threat? Are they willing to learn how real it is?
So this was the defense part. From the aliens perspective trying to hack the ship would be crucial. Wearing full space suits with self repair capability in addition to armor is required. Some form of mobile acceleration bracing would help. Your aliens anatomy could be beneficial there. Shooting holes through everything and advancing in hard vacuum would help against a lot of shenanigans. Using decoys as a front line helps the invaders to figure out how many traps there are. But the defenders will adapt to this tactic soon. Coming in from as many angels as possible will break the defense effored, especially if it improvised. Fully disabling the outside weapons and systems of the ship before bording helps, as well as slagging the enemys drive to prevent funny acceleration games. Using scanners to figure out whats behind the walls will work against traps. Rushing production hubs will be essential. Rail guns could be used to depressurise the enemy ship and take out stratigic targets.
There are numerous tactics there. The aliens challenge is to figure out what to do first. For example taking out all hull weapon and checking for traps costs time. But they want the production hubs offline as soon as possible. Additionally their honor code could slow them down too. After all isn't it cheating to surgically destroy strategic targets, which whould have to be taken at a high blood price otherwise?
As for my weapon of choice, it would be an inwards pointed ship to ship rail gun. My preferred conventional weapon would be a flamethrower, either modyfied to throw plasma or normal napalm with some thermite equivalent as a special sauce in it. A flamethrower does need less aiming than a shotgun and there is a good chance that your alien hesitates to charge through a corridor filld with an inferno. A few claymores would be nice as well.