The choice to build a city atop a mountain is governed by the balance of defense, resources and proximity. With a nod to tradition.
(Note: my answer is addresses an entire "city ON A mountain." Not "located near a mountain," "located within a mountain range," nor "nestled between mountains;" not a "castle on a mountain" with a city nearby below; not a city "on a relatively large hill." However, in any of these cases, all of the conditions below still apply, including the excellent answers and comments of the other esteemed contributors.)
Having an entire city situated upon a mountain, plateau, or mountainous hill is unusual enough to merit an analysis of its attributes.
Defense: Remote proximity from potential enemies, greater field of vision for surveillance, height advantage in combat. Safeguarding the nearby trade routes (roads and waterways) by being close enough to oversee them and to quickly deploy troops when necessary. Proximity to natural barriers that prevent enemy entry (rivers, seas, mountain ridges/cliffs, canyons) and to the structures that allow trade access despite these barriers (bridges, harbors, mountain passes, switchbacks). The nearby natural barriers protect the city, while the city protects the nearby trade structures.
Resources: Starting with food and water. A small town may hunt and farm the surrounding foothills, but it is difficult. Therefore in any larger settlement trade with nearby communities is also essential, and a proximity with them is also imperative. Cities built within mountain ranges are often in a valley bowl that is arable.
One acre of crops can feed up to 20 people for a year, if it is managed extremely well. See: https://www.quora.com/How-many-people-can-be-fed-year-round-off-of-one-acre-of-crop-growing
In colder climates the water can be collected from wells fed by snowpack runoff and natural aquifers in the mountain, but in Mediterranean or desert climates (near sea level elevations) the wells would have to extend down to the water table (sea level) just to fill a bucket. In these arid climates the majority of water was delivered by porters who trekked daily to retrieve it. This again requires a practical proximity. As a secondary system Masada had cisterns to collect rain. A mechanical water delivery system is plausible, but they tend not to work in the real pre-modern world. It is far more likely to have discovered volcanic activity producing constant hot springs near the top of a mountain.
Mining resources are a strong possibility, but not feasible without trade established among other cities, which again requires either proximity or waterway transportation. In this case it is more likely that the town sprung up around the mine, to house the families of miners, and businesses to cater to the families.
Religious Directive: (@AarthewIII - ingenious!) The deity/prophet indicated that this his desired location, holy ground, the birthplace of the deity/prophet, the burial grounds of ancestors, the place to make a stand against the encroaching enemies of the faith. Or simply tradition. (Some would argue that the deity aims to protect his flock, via all the wisdom given above.) The deal is: if the location of the city is not defensible and sustainable, then the next generation does not survive to carry on the tradition/religion.
To ensure we are defining like attributes here is a quick chart with notable "mountain cities," as defined in the chart:

And because some cities on large hills are very interesting, here is a chart of "hill cities," as defined below:

(Any good suggestions made in the comments below will be considered for the next update.)