I once lived in Idaho's Silver Valley, wherein are many, many mines.
Shallow mines (like caves) are cool, not cold. Because they are only cool, the power needed to pump something (e.g., water) into the area to vent heat to cool the surface would likely exceed power needed to cool by actual refrigerant. In other words, it's not a cost-effective solution, whether it works or not.
The deep mines (say, over 500 ft deep), on the other hand, are HOT. The deeper you go, the hotter it gets (not from magma, but from pressure). The mine companies must spend a considerable amount of money cooling the mines so miners can work.
Therefore, you can't effectively cool a building or settlement on a hot surface, but you could easily warm a building or settlement on a cold surface. Consider this simple plan for a deep mine (source, "working face" is the bottom of the mine):

Note the upper left-hand area where the excess heat collected from cooling the mine areas is utilized in various ways.
HOWEVER
You can't simply do this with an air exchange. Remember the pressures involved. Yes, heat rises, but nowhere near fast enough for many reasons, one of which is increasing air pressure as you descend into the mine. Like the Earth's atmosphere, the air is warm at the bottom where it's thick and can hold lots of heat, and cold at the top where it's thin and can't hold as much heat. Therefore, from the perspective of realism, you can only heat your colony through the process of removing heat from the mine as real mines do today.
ONE MORE THING
No mining company in its right mind would leave the mine shaft exposed to human habitation. In other words, it wouldn't be enclosed for any reason. Let me introduce you to the idea of an "air blast."
Air blasts occur for a variety of reasons, but one of the worst is that a big chunk of rock face at the bottom of the mine shifts. It needn't shift by much. Sometimes only tens of feet, but you need to understand the astronomical inertia involved along with deep-mine air and rock pressures. A massive amount of energy is released and it's following the path of least resistance: the air channel to the surface. The result is an air blast.
Air blasts kill miners. I could hear the air blasts from deep mines (2,500+ feet) on the surface 30 miles away. An air blast would obliterate anything enclosing the mine head. They represent, quite literally, the world's biggest cannons.1
So, no living near or enclosing the mine head. Nope. Nosireebob. Anyone with mining experience (or related to a miner) who reads your book would instantly recognize that as unrealistic.
1 No, you couldn't actually fire a round via an air blast. The weight of a munition round large enough to fit in the mine shaft would be too heavy for even an air blast to push. Frightening as they are, they can't be used as ranged weapons — just in case you were wondering. Physics be a harsh taskmaster, yessiree.