The late medieval period was characterized by the movement away from full plate armour and combat swinging slashing/chopping swords and crushing/penetrating maces and hammers, toward unarmoured combat, armed with thrusting swords such as rapiers and foils, and light cut/thrust weapons such as sabres.
As Tom has pointed out, when using a slashing sword, whether it is a broadsword or a katana, a long stroke is required to cut through armour, and even if the opponent is 'unarmoured', unless the blade is razor sharp, even ordinary clothing - especially if made from silk, as would be the case for samurai - could prevent serious injury.
In such a case, as Tom has pointed out, even to an indifferently trained swordsman, even if an opponent's blade was invisible, that opponent's movements would be declaring their intentions quite well enough to defend against them to a reasonable degree.
However...!
The European movement toward unarmoured combat - that came to pass due to the rise of firearms in battle that made all but the heaviest, most expensive armour effectively useless - led to the invention of thrusting swords.
These weapons were designed for sheer speed - life or death was measured in fractions of a second, inches of movement and surprisingly little force. Three inches of extension and 100g of pressure from the point of a thrusting sword against an unarmoured opponent was the difference between life and death, when delivered to a vital area.
When fencing with western thrusting swords, the difference between a successful parry and an unsuccessful one could be a matter of a few inches of movement of the tip of the sword, probably no more than 4", possibly less.
As someone with training in modern fencing, I can say that being able to see the position of the tip of the opponent's sword is of vital importance - my instructors made that quite clear. Since the sword's tip need only move 3 or 4 inches in order to avoid an opponent's parry, and the sword may be a metre long, very little movement of the hand or arm is needed in order to achieve that movement. Since one need only apply around 100g of pressure in order to penetrate clothing and flesh with a needle-sharp blade, and need only penetrate the opponent's body to a depth of 3-4" in order to cause a potentially fatal injury, the only obvious part of an attack is the thrust itself, since there is no magnification of movement caused by angles over distance.
Since a fencing blade need only be held very lightly, a moderately skilled fencer can conceal the slight movement of his arm and hand with even slighter movement of the fingers. This is what makes being able to see the position of the tip of the opponent's sword so vital.
An invisible sword would be of no use in a formal duel. No honorable second would allow the use of such a weapon (as they take charge of the weapons until it is time for the duellers to take them up), and somehow managing to use it would instantly brand the wielder as a person completely without honor, and likely lead to their being quietly stabbed in the back in some secluded location at a later date.
However, in an undisciplined street brawl or upon a battlefield, while an invisible sabre or katana would be merely somewhat disconcerting to the opponents unless they were untrained, an invisible rapier would be utterly terrifying even to the most highly trained opponents. The wielder would, with only a little practice, be easily able to handle the sword's invisible nature and parry an attack, but the opponents would be guessing blindly, quite literally, when it came their turn to parry, and with only a little skill, a simple disengage against a lucky parry could reverse the situation. Even master fencers could fall quickly to an indifferently trained opponent wielding such a weapon. An invisible rapier might allow an indifferent fencer to prevail against odds of 2:1 in all but the most unfavorable situations, and in favorable circumstances, they might stack up the corpses of their opponents in piles too great for their comrades to climb across. Were that to happen, no doubt archers, arbalestiers, musketeers or even slingers would be called so tha the swordsman with the invisible blade could be shot down from a safe distance - if any were available - or an armoured swordsman could be called, against whom an invisible blade would be at even more of a disadvantage since the wielder of the invisible blade would not be able to aim for weak points in their opponent's armour as well as if their blade was visible. Otherwise, one man with an invisible blade and a narrow passage to defend could cause an entire army to retreat, as long as their endurance held out.
Fencing is surprisingly exhausting. The constant movement is a good cardio workout, so as long as opponents were willing to keep coming, even after the comrades who preceded them had all fallen, sooner or later, the defender will become exhausted and will be unable to continue defending. If they have any sense, they'll retreat before that point, and if defence of their position is a matter of life and death, having 2 to 5 comrades, all of whom have practised with the invisible blade, could allow the defenders to hold off an army of unarmoured swordsmen indefinitely. They just need to hope that their assailants don't think of throwing rocks...
