The teleportation spell converts "mana" energy into the potential, kinetic and compression energy needed for the teleported mass to achieve equilibrium with the target destination. If the target requires LESS energy, the remainder is converted to wild mana, heat or radiation that is released at either or both ends.
In this scenario, mana is some kind of low-entropy energy that allows mages to manipulate other systems. The entropy is what makes the spell irreversible. Mages can't recharge their powers by teleporting down a few floors, as the energy returned is more chaotic (or in scientific terms: KABOOM!).
Teleportation to unknown or far away destinations would be quite hazardous in this system, as it might take all of your mana (and leave you stranded halfway) or cause a giant explosion. For short range travel, mages would likely establish a list of favorite destinations and memorize the energy requirements from a range of altitudes and distances. Included in those budgets would be the mana cost of protection against the waste heat.
For longer range travel, specially prepared teleport rooms would be necessary, equipped with either a mana focus or energy dissipation mechanisms, depending on the destination. Personal protection for the travelers is also recommended.
A downside of this would be the potential for destruction by suicide attack.
To avoid infinite-range-catapult magic, add the condition that momentum changes cost energy too. Unlike energy, changing it is always a cost. On an earth-like world (i.e. a spinning globe) that means momentum differences increase with distance because the angle changes. Teleporting to the opposite side of the planet would mean reversing the momentum completely, costing a prohibitive amount of mana. And no, you don't get hypersonic rocks that way, as the teleport won't work unless the object ends up "at rest" relative to the destination. Something about reintegration failing when moving too fast.