There are plenty of toxins that take quite a bit of time to harm humans, or aren't even harmful at all in reasonable doses,because they are more immediately harmful to other creatures. Examples include chocolate, tobacco, poppies, marijuana, willow, peppers, and coffee.
A poisonous plant's typical targets are insects and caterpillars, so if it's not poisonous to humans, the evolution of the plant is hardly affected -- though being less poisonous to humans can tend to keep us from clear-cutting it away (as in the case with poison ivy, poison oak, etc.), and in the plants that I've listed, having a secondary effect on humans that is not overtly toxic, and possibly beneficial (willow bark is the basis for aspirin), humans may actively cultivate it.
In some carnivorous plants, especially certain species of pitcher plants, there are some traps that are inactive most of the time... such as nectar-bearing flowers where only about a third are slippery at any given time. This attracts more insects overall, as there is a potentially high benefit to individual insects that are lucky enough to pick the right flower, and a consistent benefit to the plant, in the form of the micronutrients that insects bring. (The main macronutrients for plants, of course, being water, CO2, and the sunlight to process them.)
If your plant is in a region where the soil lacks key micronutrients, carnivorous plants may have a poison that strengthens or weakens over time, so that animals may get addicted to side effects and harvest regularly, then when the plants needs that nutrient, it could increase the potency of its toxin, hoping that some animals die above its root system.
If your village has been eating the same berries frequently, you might not suspect a plant when it only rarely has a dangerous phase. It would take most people several cycles before they figure it out.
And, of course, the mutation that causes the slow acting toxin could be completely unrelated to either the survival or virility of the plant... Prions are a class of proteins found in all mammals, usually specific to certain families of species (primates, canines, felines, rodents, etc.) A prion that mis-folds can start causing damage to various cells, and would cause other related prions to also mis-fold -- essentially a viral infection without a virus, and without an immune response. Prion diseases can be dormant for decades and, besides getting it from just bad luck, prion diseases can also be transmitted by eating food contaminated with that prion. (Cannibalism is bad. Bury your dead away from your farms.)
It's possible (unlikely, but most certainly in the realm of believable Sci-Fi) that a plant can start producing the dangerous form of a prion that humans use. In which case, in a few years, the people who eat from it will start showing signs of prion diseases. This mutation wouldn't help or hurt the plant at all, it would just make sure that any primates that eat from it will die between 2 and 20 years from that day.