If I'm understanding you correctly, you're drawing an implied link between feudalism and agriculture, in which the peasantry or serfs need to be inefficient agricultural workers for the institution to survive.
However, this depends entirely on the tradition and context one is discussing. In western Europe serfdom didn't survive the black death. This owes to two factors, firstly that in this region there were upper, lower, and middle classes. The middle class; freemen or yeomanry in an English context, had legal rights and could own land.
This was very important, as it allowed families to accumulate wealth and status down the generations, eventually joining the upper class... perhaps even seeing their grandchildren able to woo royalty. This meant that the mass death caused by the plague rose the cost of labour and allowed the middle and lower classes bargaining power, especially as in England there had been basic constitutional rights granted, restricting the monarchy - monarchs and nobles couldn't just enslave or murder freemen going about their business. They had negotiating power owing to their legal rights as free men.
The situation in eastern Europe, especially Russia, was very different. While the English king agreed to Magna Carta in 1215; thereby limiting his power and increasing the rights of nobles and free men, the lesser known Sobornoye Ulozheniye drafted in 1649, led to greater restrictions on social mobility in Russia.
The latter code merged the lower classes into serfs, making them property. Not only this, but only the nobility could own land, and the serfs were heavily restricted, needing internal passports to travel between towns as they did not have any right to travel.
One was not a serf because they worked the land. There were many "house serfs" who served their lord's estate as household servants. There were many other roles serfs played, from traveling entertainers to soldiers, and all the time they remained property. Their noble masters could do what they liked to them, as the Tsar could do as they liked with anyone.
Increasing agricultural efficiency does not change that, because feudalism is a legal principle and structure. Nobles could put their serfs to work as they pleased, including factory work. See the likes of "The Condition of the Serf Workers in Russia's Metallurgical Industry 1800-1861". Factory serfs did not have life any better than their farmland peers, in fact the argument goes that they had it much worse, proving that technology does not in itself provide social emancipation. Keep in mind that by then, still about half the country regarded the Tsar as a living God, and the aforementioned legal code survived until 1849.
Serfdom as an institution was only abolished in Russia in 1861, but for many the reality remained the same for decades to come. So you need a lot more than just increases in agricultural productivity to have a meaningful effect upon society... much less the abolition of feudalism.