Edit: (September 18 2021) So far there are are many good answers that address why a group of guard monsters or unaffiliated group of humans might have a finite aggro range.
However these answers are unsatisfactory, because they do not attempt to explain the example where the heroes can attack a castle and clear it room-by-room, without having to fight all the guards at once.
In this case the guard groups are affiliated with each other. In reality I would expect a command structure with a plan for what to do in case they are attacked. I would not expect them to stand idle while the heroes murder their colleagues at the other end of the room, and wait for their turn at the slaughter.
Often in video games (and to a lesser extent tabletop games) the enemies have finite aggro ranges. That means for example, even on a featureless flat plain, the Man-Eating Troll will ignore the heroes before they come within 60 feet. Only then does the troll attack and try to man-eat the heroes. If the heroes later retreat to outside the aggro range, the troll forgets they exist and returns to its original position or patrol route.
In practice this is to allow adventures where the heroes take on a much larger force by fighting them in small groups. For example to take over a castle, first they fight the soldiers at the Gate House. Then they fight the soldiers at Ramparts East. Then Ramparts West. Then the Guard Tower. . . and so on until every room of the castle is cleared separately, and the final encounter is with the king and his royal bodyguards.
This requires great suspension of disbelief, since attacking the gatehouse should realistically alert all the troops in the castle, either by raising the alarm or the noise of fighting drawing soldiers from elsewhere. Soon our heroes will find all the soldiers in the castle on top of them.
An even sillier example -- in large rooms our heroes can often fight and kill the Manticores at the entrance without drawing the attention of the Basilisks at the exit. Even though the heroes can see the Basilisks, the monsters are either unaware or indifferent to the heroes, since their aggro range is smaller than the room.
From a world-building perspective, what would explain this strange behaviour? The goal is to keep as much of this strange video-game features while making as few changes to the real world as possible.
For example we could declare that all the monsters and enemy troops are in fact golems that are powered by crystals hidden below the floor. They simply fall down dead if they stray too far from their spawn points. This also explains why they fight to the death. They are non sentient and so lack a self-preservation instinct.
However this solution falls down in that (a) it is is a radical departure from the real world and (b) it does not explain why the golems are so common, when they are less effective at defending the castle than living beings, and presumably more difficult to acquire.
So what is the optimal tradeoff between modifying the real world as little as possible to produce as much strange video-game logic as possible?