It is indeed realistic, mainly because most humans fear death, and it probably happened plenty of times in history.
We can't really take sport fencing or computer game mechanics as a basis. If they were fighting with training gear fully protected, or if they were mindless drones where the master swordsman differed only in physical strength and slightly better reflexes, then yes, even just 2 or 3 lesser skilled opponents would have a great chance of defeating a single more skilled opponent.
First, let's assume the fight doesn't start with the master surrounded, because that would reduce his chances enormously. One of the most important factors would be how the fight starts. Most people don't realize instantly that the situation is a life-or-death one, and they will lose precious seconds in the confusion.
1. Most people lack the mindset and the intent to kill
There was a research shortly after World War 2 which showed that a huge percentage of soldiers were not even firing into the direction of the enemy, and even those who fired, very few of them were aiming at individual soldiers, most were just firing wildly in the general direction of the enemy. If the opponents are just soldiers pressed into service, or they are bandits fighting for loot instead of a cause which would motivate them to kill and to risk being killed, it would look like most bar brawls are: they are more into scaring their opponents away or dominating them, than outright killing them.
2. Most people fear death
Those of you who ever sparred, practiced sport fencing, or just LARP, would you go into the fight with the same eagerness if you knew it was to the death? Probably all of you were hit at least once during practice. Imagine it was deadly.
Technically, if several less skilled swordsmen lounged forwards at the exact same time, one would be surely killed by the lone master, but the others would manage to kill him in return. But are they coordinated that well? Probably not. And who is the one who wants to sacrifice his life to go first? Especially if the attackers are bandits, who don't fight to protect their loved ones, so they don't have anything to sacrifice their lives for...
Probably, as the fight starts, everyone expects someone else to strike first, and a precious few seconds will pass during this time of indecision. There can be at most 4 or 5 in the front facing the master directly (any more and they would just get in each other's way), the rest would be behind them. Now as they wait for someone else to strike first, the master swordsman quickly finds the one or two whose guard is the most vulnerable, and stabs them before they can react. They don't fall to the ground like sacks of potatoes, like you can see in movies or video games. They will writhe violently on the ground, screaming in agony at the top of their lungs, with blood spattering around. Some would vomit at that sight, some would run away, and many of the rest would be frozen for a short moment, allowing the master swordsman to strike again.
At this point those who are still uninjured, would probably fear for their lives and just wildly flail around with their weapons, not really eager to close the distance. They will be uncoordinated, allowing the master swordsman to engage them one or two at a time.
4. Fight-or-flight reflex.
It takes a lot of training to overcome basic instinct. If the fighters are just averagely trained, in a life or death situation it can happen that all their training and all their techniques go completely out, and they are back at basic animal instincts.
This is what makes a great difference, especially when combined with the fight-or-flight reflex mentioned above. Fighting against armor with a sword requires very elaborate techniques and great precision to stab at the more vulnerable spots of the armor, and there are complex techniques for getting into a position to be able to perform those movements. You can't defeat metal armor by just wildly hacking at it with a sword. If the fighters revert to basic instinct, they will not be doing much damage against someone who wears armor. If the master swordsman is really well trained, his mind will not be affected, and he will still be able to perform the techniques needed to defeat his opponents even if they wear the same quality of armor.
Even if the attackers are individually not too bad at fighting, they might lack the skills and the practice of fighting in a coordinated way, and they might just get in each other's way.
Here is a great video demonstrating many of the issues mentioned above: https://www.youtube.com/watch?v=7s7KfetTixQ