How big was the asteroid? Why are there only a few survivors left?
If it was because the impact itself killed most people off, the asteroid was very large and has created significant problems well beyond what would be faced if it was a small asteroid which triggered human-caused mass death (religious extremism bringing worldwide violence, fear causing nations to desperately grab for resources causing WW3, etc).
The larger the asteroid, the more extreme the effects. Hot debris could cause global firestorms, resulting in a low-oxygen high-CO2 atmosphere - a couple of years of darkness and serious acid rain killing whatever survived the firestorm and killing off aquatic life too, followed by a hot earth as tiny life regrows. Big enough to wipe the ozone is big enough humans won't survive.
It will be difficult to thread the needle on big enough to kill most people directly, but small enough for scattered survivors to make it (presuming there is even any area in-between - I suspect big enough to cause eventual extinction comes before big enough to kill most people on impact).
If a smaller asteroid, the impact is more of a narrow regional problem with the exception of reduced sunlight. This isn't going to result in glaciers retaking to world, just relatively normal wintry temperatures holding year round (spring comes in a year or two instead of a couple months). Massive food shortages (no natural growing season for a year or 3) will cause global disruption and worldwide starvation, but some will be grown in greenhouses with supplementary grow-lights, probably in addition to aquaponic systems if you can get them set up (I foresee diets of bugs and farmed algae).
If you want scattered survivors after an impact, your best option is probably to go with a small asteroid, whose global effects are really just limited to not being able to grow food for a year or two, followed by human-caused disaster. Religious revivals (it was the angry fist of god) leading to mass warfare as the fanatics purge the unbelievers, plus skittish governments starting panicked wars of desperation to control resources, and general social panic being the source of collapse leaves civilization collapsed but humans not extinct. Nuclear exchange in the aftermath of impact would give you fallout as a reason to stay underground, as well as defensible shelter from roving bands of the various religious inquisitions.
People need sunlight and that is going to be difficult enough from the ash clouds - every moment of daylight should be spent outside, and even then they should be keeping a power plant going for greenhouse grow lights and supplementary lighting for vitamin D generation (especially to keep children from getting rickets). There better be a very good reason to stay indoors, and no asteroid is going to give you that without also spelling extinction.