Mountain goats, and some other mammals, are exceptional climbers. They can climb slopes at greater than 60 degrees. Unfortunately they are too weak to carry humans or goods without serious injury risk. This lead me to wonder, is it possible that a beast sturdy enough to bear a human could be mountain dwelling and climb slopes of >60 degrees or is there a limit based on weight, size, shape or muscle that means a beast strong enough to carry a human could not climb these slopes?
I think you want to move away from goats and look at monkeys. Monkeys and even the great apes carry their young on their stomach or their back. They can still climb, sloths do too btw. And Giant Sloths existed. They weren't tree dwellers but they could be changed into climbing mountains.
As John mentions in his answer, climbers without claws or hands rely on their center of mass to retain balance. Adding a moving rider to that is a recipe for disaster. So hooved animals are a poor choice here.
Your big problem is if you start adding and taking away weight from a climber you throw off its balance, a rider will drastically shift their center of gravity and widens the animal, making steeper slopes much more difficult. breeding for size is not an issue, but drastically shifting the center of gravity and widening the animal is, especially for a goat like climber. put 50 degrees as your max and you will have much more success.
A takin might be able to do it.
Takins are sturdily built. Males takins can weigh 600-700 lbs and are 4 feet at the shoulder, which is comparable to a horse. They live in mountains and can negotiate steep slopes.
I think a rider who does not want both of them to pitch over backwards when going up a steep hill could switch and ride underneath the takin, Odysseus style.
You might be surprised at how steep a grade a horse & rider can climb. I've ridden up & down slopes pretty close to 45 degrees myself, and I'm far from an expert. Part of the trick is shifting your weight to keep the horse in balance. You don't just sit in the saddle: in even a moderate climb, you shift your weight to balance over the front legs. On a steep grade, you can be almost lying along the horse's neck. On descents, just the opposite: you lean back to keep the weight on the horse's back legs.
For examples, do an image search on e.g. "Tevis Cup Cougar Rock", and you'll find plenty of pictures like these: https://mfthba.com/wp-content/uploads/2014/11/cougar-rock-300x212.jpg http://www.horsesinthemorning.com/wp-content/uploads/2015/07/Karen-Chaton-Bo_Cougar_Rock_2013-1024x820.jpg
The real problem (other than keeping your saddle from slipping) is the footing. Just as with humans, loose footing can make a slope difficult or impossible, not to mention dangerous for horse & rider. I think, from the few times I've seen mountain goats in action, that they are not actually climbing a slope, but going from one small level ledge to the next.
As a general rule, mountain climbing mammals tend to be small, the largest being goats, donkeys, and bighorn sheep. They live in the mountains because they have found a niche with an unused food source, and no predators.
Anything larger either has to be a roaming herbivore that may not find the forage it needs to thrive on mountain sides, or a carnivore. The problems of trying to ambush prey with little concealement and carrying it off, all without falling off the mountain yourself, preclude large carnivores from being mountain dwellers.