Assume:
- as many individuals per generation as needed to show all possible phenotypes
- the desired trait depends on a single 1-1 phenotype to genotype relationship - there's two genes and they each correspond to a single phenotype when expressed A-small fruit and b-big fruit
- one gene, undesirable, has complete dominance over the other, the desirable trait. That is, if there's ANY A in the offspring, it will show small fruit. Big fruit only shows when both genes are b.
- each progenitor (first generation parent) is a mixed genotype (therefore would have small fruits, but carry the potential for making big fruits)
- and there's no significant genetic mutations (there's ALWAYS mutations, most don't make much of a difference)
- complete knowledge and control over pollen exchange
IF all of the above is accepted, then you can change the phenotype to big fruits that breed true in only one generation, because all plants that exhibit the BigFruit phenotype would have double recessive, bb, genes. None of the above assumptions are very likely, so this result is pretty unhelpful except to show how simple simple would have to be in order to have a clear and consistent answer.
As others have said, luck, perception, scientific knowledge, the genetic complexity of the desired trait, environmental factors contributing to gene expression, methods the plants use for moving genetic material (eg pollen), the people's ability to manipulate the plants and environment (not like in a lab, but like, can you keep bugs away? can you block the wind, etc) to control the pollination, the people's knowledge of how this genetic stuff even works - do they know what pollen is? do they actively intervene in pollination, or are they selecting seed based on whether the plant exhibits the desired trait? Can they vegetatively clone the result once they have one they like? Sometimes crops become crops specifically because they respond to breeding efforts.
The answer is somewhere between 0 and impossible. There are plants and animals that are, as far as we know, undomesticatable. Some plants (peas) are used to teach genetics and breeding due to the relative simplicity of some genotype/phenotype relationships, their usual tendency to self-pollinate (making controlling pollination relatively simple) etc.
tl;dr - This is an incredibly complex question. It depends on so many factors that it is unanswerable except by "one or more generations, if it's even possible".
Now, if you have a situation where you want it to take, say, 10 plant generations, and you want to know some things that would make it so that that's coherent with the traits of the plant, that might be more answerable.
But, with only the information you have provided, and my knowledge (I work with horticulturalists and germplasm specialists, but am not trained as such myself), I do not believe there is a "correct" answer.
Hope that's helpful, sorry it's not a more concrete answer.
These resources, however, may be interesting to your worldbuilding:
This one allows you to search crop plants by desirable traits and, for many, get a genetic lineage for them: https://npgsweb.ars-grin.gov/gringlobal/descriptors
This one allows you to search wild crop ancestors and find out what traits have been introduced when bred back into domestic lines: https://npgsweb.ars-grin.gov/gringlobal/taxon/taxonomysearchcwr
Sources: years of trying to make sense of working between genetic researchers, germplasm resources, and farmers on the ground trying to solve crop issues with more training in communication than biology.