In my answer to the question, How would the design of a habitable underground fortification differ to that of a castle?, I described in some detail the construction of an underground fortification.
An intelligently designed underground fortification won't be as simple as a cave with gated entrances. The entrances will be designed to be actively defended, so if the raiders attempt to force entry or even blockade them, they will be in range of the defenders' defences. An assault on the gates would quickly turn into a massacre for the raiders, and if they have any sense, after the first attempt on the gates, they won't try again.
Castles are force multipliers. They allow a small number of defenders to engage a larger number of enemies. For surface fortifications, the usual multiplier was ten - for each defender, the attacker needs ten men to achieve nominal parity. For an underground fortification, the multiplier may well be higher - perhaps twenty to fifty or even more, depending upon the nature of the fortification and the geology in which it is situated, twenty if it is in soil or gravel, to fifty or more if it is excavated into hard rock. Therefore, the defence force of 400-500 men could be expected to hold off an enemy numbering between 8000 to 25000. The 800-900 'vikings', having run once into the meat-grinder that a well-constructed set of defences could be, wouldn't stand a chance. The best they could hope for is to escape with their lives. Picture this:
The Vikings locate the entrance to the underground kingdom, and prepare a battering ram from a convenient large tree in order to take on the wooden gates set into the cliffside. They advance under their shields as the defenders rain arrows down upon them. The fact that the arrow-loops on the cliff face are entirely enclosed makes counter-attack difficult - an arrow must be aimed to pass through the loop, and siege weaponry munitions are too large to pass through the loops, the only hope is containers of burning oil, but if those were to be used, the oil would burn the siegers more surely than the defenders.
On reaching the gate, the ram is swung, and despite losses, the gates are breached. A few defenders flee down the tunnel and vanish around a corner- those that aren't killed. The raiders follow, only to find that just around the corner is another gate, but this one isn't wooden, it is made from tens or even hundreds of tons of stone, rolled sideways into place. The ram cannot be employed - the passage is too narrow to turn its great length, and no application of strength can roll the gate aside - the defenders have placed a big wooden wedge behind it, and the raiders might as well be trying to push a stone up a mountain.
Then, when the raiders have swarmed into the entry tunnel, they discover why it is so long and lightly gated at the surface - the whole thing is a death trap. The defenders open hatches in the roof and rain down burning oil, rocks and any other deadly substances they have. A little forethought by the defenders would provide grated vents near the main stone gate and any fire within the tunnel would be fed air from there, the downwards slope of the tunnel acting like a chimney to emit the smoke from the flames from the gate, and the draft would fan the flames.
If the defenders are especially nasty and have had plenty of time to work on their defences, they could close a second, hidden, stone gate just behind the remains of the wooden gates that would trap the raiders, making their escape impossible while they burned to death. Defences like this could easily kill hundreds of men, and the raiders only have hundreds Then the defenders could open the outer stone gate again and dare the attackers to advance inwards again over the charred corpses of their comrades.
So, the raiders have been presented with a fortification that they cannot defeat. They could try to besiege it, erecting their own fortifications around the gate to prevent the defenders from resupplying, however, there is the matter of the second gate.
The second gate would be similarly equipped with defences as the first, but in addition, it would be hidden. There would be observation/defence posts overlooking it so that the defenders could see if it had been discovered or if any enemies just happen to be nearby, but it would be placed in an area too narrow to make a good camp, but the gate would look like nothing more than a featureless rock face.
The second gate would allow the defenders both to resupply and to exit their fortification unseen, at night, when they would engage in guerrilla warfare against the raiders, inflicting atrocities upon them at night and whittling down their strength until they have no choice but to flee.
By engaging in guerrilla warfare, a small force can take on and defeat a larger one piece by piece, or force them to retreat to their own fortifications, making their own resupply far more difficult, and should the raiders do that, they are doomed, as the defenders have better supply.