A Dead Planet:
Your world is very old, and drifted through interstellar space for a long time before getting captured by the star it now finds itself around. As such, the core of the planet has cooled to the temperature of background space - very cold. The surface has slowly warmed over the ages, but deep down the world is frigid.
Ancient ice ages:
Your region was very cold for a very long time. Perhaps your cavern even used to be a buried glacier and was gradually melted as the surface warmed. Your cave is thus effectively an ice box, with the surrounding rock keeping it cool.
Intense long seasons:
Your world has regular and very long cycles of long seasons - like decades at a time. The world is currently in a warm phase, but not that long ago it had sustained bitter cold. So the thermal mass of the ground gets cold very deep down, then takes a similarly long period to warm back up.
Evaporating Gasses:
Someone long ago used this cavern, and more beneath it, as storage tanks for vast amounts of ammonia. But the folks who did so are long gone, and the ammonia gradually evaporated (chilling the rock). Further, ammonia from below continues to evaporate and act like a huge refrigerator. There would likely be very high amounts of ammonia gas leaking out of the caverns, making the air rather dangerous (poisonous and flammable, the degree to which is up to you). This would also provide a reason why no one has exploited your caves before - it was simply too dangerous to do so.
Depending on the setting, this could be the reason for your people going there - ammonia could be a valuable resource, and they want to exploit the remaining ammonia in the lower caves for lifting gasses, refrigerant, fertilizer, chemical manufacturing, or water purification.