I note that popular images of the airless moon showed tall, steep, and jagged mountains until photographs in the 1960s showed the hills were low, smooth, and rounded.
Telescopic photographs of regions of the moon were often taken when the Sun was low in the Lunar sky, thus making long shadows of the mountains and crater walls. Astronomers could calculate the true heights of mountains and crater walls if they knew what angle the Sun was at when the photos were taken, but people who just looked at the photos assumed that the mountains were very tall and steep.
The factors that made the Moon airless also made it geologically inactive for billions of years, meaning the mountains on the Moon were billions of years old. Even though those billions of years old mountains weren't weathered by windblown dust or by precipitation, they were weathered by billions of years of temperature changes, harsh solar radiation, bombardment by charged particles in the solar wind, and by a slow steady bombardment by micrometeorites and occasional large meteorites.
Of course a small world like the Moon could have been terraformed by space travelers when it was young and given a breathable atmosphere when its mountains were still tall and jagged. I have read that if humans terraformed the Moon by giving it an atmosphere that atmosphere would escape in a thousand years. I read once, in a story probably co authored by Arthur C. Clarke, that the Moon was terraformed and the atmosphere was kept from escaping by a layer of nanobots that were attached together and covered the entire atmosphere and bounced back all molecules headed upwards.
For your planet to be habitable for humans, it must have the right proportion of various gases in its atmosphere and a tolerable total atmospheric pressure. The vital oxygen in Earth's atmosphere was produced by plants, and it took billions of years to do so naturally on Earth.
So the planet should have a surface gravity and escape velocity high enough that the vast majority of the oxygen has not yet escaped from the planet. Which means that if the habitability of your world is natural and not due to highly advanced terraforming the surface gravity can be considerably less than that of Earth but should be greater than that of Mars, for example.
I also point out that humans need only the oxygen in Earth's atmosphere, since they only need to breath that oxygen for breathing, and only need small amounts of carbon dioxide and nitrogen to support plant life and small amounts of water vapor to keep the air moist enough. That a breathable atmosphere could be a lot less dense than Earth's, which might reduce the rate of weather erosion. Though a highly reactive mostly oxygen atmosphere might reduce rocks to rust and dust and crumble mountains.
Someone may point out that the surface gravity and escape velocity of Titan - 0.14 g, and 2.639 kilometers per second - are much lower than those of Mars - 0.376 g, and 5.027 kilometers per second - and lower still than Earth's, 1.000 g and 11.186 kilometers per second, and yet the atmospheric pressure on Titan is many times as great as on Mars, and even a bit greater than on Earth.
One reason for that is that Titan is much farther from the Sun than Earth is, and so it is much colder than Earth, far too cold to be habitable. So the temperature of the molecules of gas at the top layer of Titan's atmosphere, where gases escaped from the atmosphere into space, is much lower and so the molecules are moving much slower, and a smaller proportion of molecules reach escape velocity and zoom off into space.
If Titan was moved to Earth's distance from the Sun, it would be hot enough at the surface to be habitable, and the gas molecules at the top of the atmosphere would be about as hot and move as fast as the gas molecules at the top of Earth's atmosphere. Thus Titan's atmosphere would escape into space much faster than it does.
Observations from the Voyager space probes have shown that Titan's atmosphere is denser than Earth's, with a surface pressure about 1.45 atm. It is also about 1.19 times as massive as Earth's overall,[34] or about 7.3 times more massive on a per surface area basis.
https://en.wikipedia.org/wiki/Titan_(moon)1
The persistence of a dense atmosphere on Titan has been enigmatic as the atmospheres of the structurally similar satellites of Jupiter, Ganymede and Callisto, are negligible. Although the disparity is still poorly understood, data from recent missions have provided basic constraints on the evolution of Titan's atmosphere.
https://en.wikipedia.org/wiki/Atmosphere_of_Titan2
Thus until the origin and survival of Titan's atmosphere is better understood, it seems advisable for writers to avoid giving worlds as small as Titan dense atmospheres, especially dense atmospheres that have the same temperature and composition as Earth's.
There may be ways to give your world a breathable amount of oxygen without it being produced by life over billions of years during which oxygen and other gases in the atmosphere would have weathered tall mountains.
For example, the world could be an Earth like but very young planet which has young tall mountains and one large ocean basin filled with water. The world suffers from runaway glaciation and the ocean basin freezes. A giant asteroid impacts, probably in the ocean basin, and vaporizes a lot of rock and all the ice in the ocean basin. the heat is so intense all of the vaporized water also separates into hydrogen and oxygen. Almost all of the super heated hydrogen escapes from the planet into space while more of the heavier oxygen is moving slow enough to be retained by the planet and becomes an oxygen atmosphere. Some of the hydrogen is retained and recombines with oxygen to form water, and water vapor in the atmosphere serves as a greenhouse gas and keeps the temperatures warm enough for humans. Of course the impact would probably release a lot of poisonous gases into the atmosphere and you would need processes to reduce them to breathable levels by the time of the story.
As near as I can tell the tallest mountains in the solar system are: 5) Ascraeus Mons on Mars 14.9 kilometers or 9.3 miles high, 4) Boossaule Montes on Io 17.5 to 18.2 kilometers or 10.9 to 11.3 miles high, 3) the equatorial ridge on Iapetus 20 kilometers or 12 miles high, 2) the Olympus Mons Volcano in Mars 21.9 kilometers or 14 miles high, and 1) the central Peak of Rheasilvia, a crater in the asteroid Vesta, that is 22 kilometers or 14) miles high.
Two of them are on Mars, a planet large enough to possibly become habitable under some circumstances, such as being terraformed by humans in the future. But I believe that the slopes of those volcanoes are so gentle, and the curvature of Mars's surface so great, that they would be impossible to appreciate when looking at them. But cliffs at the base of Olympus Mons are up to 6 kilometers or 3.728 miles high.
On Venus Skadi Mons rises 0.11 percent, or 0.0011 of Venus's mean radius, on Mercury Caloris Montes rises 0.12 percent, or 0.0012, of the mean radius of Mercury, on Titan Mithrim Montes rises 0.13 percent, or 0.0013, of the mean radius of Titan, on Earth Maua Kea and Mauna Loa rise 0.16 percent, or 0.0016, of the mean radius of Earth, and on Mars Olympus Mons rises 0.65 percent, or 0.0065, of the mean radius of Mars. The other two bodies in the solar system, Ganymede and Callisto, that might possibly have become habitable under other circumstances, or that humans might reasonably expect to terraform in the future are largely made of ice and thus very flat.
A number of smaller bodies in the solar system have mountains whose heights are larger proportions of the radii of the bodies, but those are smaller bodies even less likely to be habitable than Mercury or Titan. The largest of them is Io, where Boosaule Montes rises 1 percent, or 0.01, of Io's radius. Io is considerably smaller and less massive than Mercury, Ganymede, Callisto, or Titan.
The tallest possible height of a surface feature is determined by the structural height of the materials it is made of and the materials below it. Once the height, and thus the pressure at the bottom, exceeds the limits of the material it will soften and flow out from underneath the surface feature which will then sink until it reaches a new pressure equilibrium.
The stronger the material, the higher it can be piled before the pressure at the bottom becomes too much. And the lower the surface gravity of a world, the higher a specific material can be piled without the pressure at the bottom becoming too high.
Thus the smaller the surface gravity of a world, and the stronger (which usually means denser) the materials it is made off, the higher its mountains can be. Unfortunately the surface gravity is usually proportional to the density of a world, so worlds with low surface gravity tend to have low density materials, and worlds with high surface gravity tend to have high density materials. Also planets with low surface gravity tend to have difficulty retaining for geologic time periods atmospheres dense enough to be breathable.
So one possibility would be a super Earth or mini Neptune type planet that lasts long enough to become layered with the densest material sinking to the bottom. So it will have an iron nickel core. Then it is struck by another planet early in their solar system's formation, and the giant explosion vaporizes most of the planet and most of the materials of both planets escape from them. But parts of the cores of the two planets remain liquid instead of gas or plasma, mostly the densest materials. The cores capture each other, orbit each other, and gradually spiral inwards until they merge in another giant collision.
The liquid nickle iron core of the merged planet gradually cools down and solidifies into a solid planet that is mostly nickle, iron and other heavy metallic elements. Meanwhile many of the gaseous and liquid fragments of heavy metals from the planetary cores remain in orbit around the merged planet, cool off, and solidify, and clump together to form larger and larger bodies orbiting the merged planet.
Tidal forces push the outer fragments farther and farther from the planet, where they may merge into one or more moons, and pull the inner fragments down toward the planet, and they will eventually crash into the planet. That will form craters and circles of ejected materials, which may be taller than those on rocky planets if they are made entirely of various metals.
And maybe a few more collisions can provide the planet with a thin layer of rocks and soil and water and an atmosphere, or maybe it will be terraformed by aliens or humans.
Another factor which affects the height of mountains on a planet will be the forces that produce mountains. As a general rule, larger, more massive, and denser planets will tend to have stronger forces to thrust up mountains as well as stronger forces to tear down mountains.
One way for a smaller body to have stronger geologic forces than it would normally have would be tidal interactions with one or more other worlds. One of the worlds mentioned above, Io, has constant volcanos because of tidal interactions with Jupiter, and also has a tallest mountain, Boosaule Montes, that is very high in absolute terms and also very high relative to the radius of Io.
The volcanic rate of Io is so high that Io would be a dangerous place to live, and might raise the surface temperature of Io high enough to be uninhabitable. On Europa, the next farthest moon, the volcanic rate is not high enough to keep Europea warm enough to be habitable.
But if an Earth or Mars sized moon orbited as gas giant planet that happened to be in the habitable zone of their star, and if that planet sized moon orbited the gas giant planet at the right distance to have the right amount of tidal heating and volcanoes, it might have a breathable atmosphere as well as a much more impressive terrain, the result of more active geology, than Earth does.
Or maybe you might write a story where tourists on the planet Garbruth see all the famous natural wonders like the Coslorm Spires, the Jarganth Mountains, the Grand Canyon of Lymfar, the Mountains of Wadmoss, the Chasm of Owo, the Rock Arch of Hawtute, The Cliffs of Klabon, and Mount Ebertast, to name a few, and come away with the impression that the planet is all vertical land forms. But flying back to the spaceport, one of the tourists does something they didn't do before. They find a view screen of the vehicle and set it to show what they are flying over, and they see the endless Plains of Jahanne.
