Let's assume for the moment that physics, biology and chemistry are not all down the pub getting smashed on dwarven sprits, even if they might be a bit distracted in your world.
When humans train in physical tasks they build muscles that work well for those tasks - white muscle for speed, red muscle for strength and endurance. The more muscle you have the better you perform at tasks that use those muscles... at least until the muscles themselves start getting in the way.
Besides the strength and speed you get from those muscles you also get a whole bunch of other effects:
- Increased energy usage
Everything you do, including breathing, now takes more energy. That energy has to come from somewhere so you eat a lot more than you used to. This means more time spent eating and more money spent on buying food... or more time hunting your next meal, depending on the setting.
- Extra Heat
The bigger your muscles and the harder they work the more heat your body generates. Human bodies are very delicate about their internal temperature so your body needs to get rid of that heat somehow. Mostly this involves increasing the blood flow to the skin and then sweating to cool the skin down. As you increase in mass your volume increases faster than your surface area, so eventually you're going to hit a breaking point where your skin area isn't high enough to provide enough cooling to balance your muscles. Either you stop before you hit that point or your move to a glacier to continue your training.
- More Waste Products
Every time you flex a muscle the body uses ATP, glucose, etc. to make the cells contract. The waste products need to be moved out of there pretty quickly or you will get to enjoy all the fun effects of things like lactic acid buildup. Your body's ability to deal with those metabolites tends to increase as your improve your fitness, but the waste products still need to be filtered out of your blood and eventually out of your body entirely. And we haven't even started talking about the effects of all that extra food...
- Bone Strength
Human bones are fairly strong, and they do tend to get stronger as you bulk up. But there are limits there too (unless you remove them) and it's possible to build muscle strength faster than bone strength. In human body builders today you still hear the occasional story about someone who bulked too fast and broke a bone during a lift. Admittedly there are probably some contributing factors like steroid abuse going on there, but it happens.
- Sheer Mass
Ever meet a really big body builder? You know, 6'5" and built like a human-shaped bulldog? Ever wonder how much they weigh in at? It's not unusual for an average height human bodybuilder to reach as high as 270lb, and some of the taller ones can get up around 300. Ronnie Coleman weighed in at about 297 lb at his peak. That's about twice the weight of the average slim guy.
Now imagine 300 lb pushing down on an area the size of your foot. OK, maybe bigger than your foot, but think it through. You'd tend to find the weak points in any floor you walked on by the simple expedient of falling through them. And when you run the impact force is much higher.
I had a friend who did competition body building, and when he bulked for competition he had trouble with normal doors. He had to turn sideways to get through an average house door. He had trouble finding a vehicle that he fit in properly without modification. Dude was big.
With great size comes great... well, joint pain for one thing. Extra stress on the heart - which I presume isn't a problem here because it's just another muscle - and other organs. Hormone imbalances from the excess musculature. Intestinal problems from the massively different diet. Damaged cartilage, tendons, nerve sheaths... there's a lot that goes wrong for big, big guys. And that's just the stuff the natural body builders suffer from. Wait 'til you see just how screwed up the chemistry is inside a 'roid head. That stuff is crazy.
Actually there's more, but I think that'll do for now. Most of it comes down to one thing: the Cube-Square law.
If you want some more ideas I suggest an online series called DeathWorlders. One of the ongoing themes is using alien pharmacology to train human soldiers to absolutely absurd size, strength, speed, etc. One of the main characters "Warhorse" is frequently compared to The Hulk in size... and he's still growing.