Unless I am very much mistaken, the difference between the moderate climate we had some 200 years ago and the last ice age was that the global mean temperature was 3°C lower during the Ice Age.
During an Ice Age, obviously there is much ice and snow, both of which tend to be bright and tend to reflect a lot of energy, preventing the ground below and the air above from warming.
So, consider a setting where, due to something like a fluctuation in the gulf stream, the polar regions are 1° cooler than average. This would likely result in a somewhat larger area receiving snow, which would create a little bit more cooling, and so forth. I think it is conceivable that this could lead to a mini-Ice-Age, at least regionally. If the spread reaches far enough to the south, the "corn belts" could be affected, and in turn affect the food supply of a much larger region.
Weather is quite chaotic and hard to predict, but I think that with a somewhat more fragile setup, it should be possible to have much more erratic seasons, simply based on small coincidences adding up.
