There are several possibilities that come to mind.
You've already mentioned one: Sexual selection. In short, they find taller people more attractive. It could be cultural, it could be hardwired into their genes somehow.
Others include the other modes of natural selection. If taller Jotnar are better at obtaining food and avoiding getting themselves killed, the Jotnar will gradually get taller over time. Ideally, there should be an evolutionary arms race between the Jotnar and some other species. Perhaps there's some predator that a Jotunn can only effectively defend itself against if the predator is shorter than the Jotunn. That way, there's evolutionary pressure for both species to become taller. Or, perhaps there's a tree whose leaves make up a substantial portion of the Jotnar diet. The tree will evolve more height, smoother bark, and fewer branches to make it more difficult for the Jotnar to climb, while the Jotnar will evolve to become taller and better climbers in order to get to those tasty leaves. This might also happen if the Jotnar are going after the tree's fruit rather than leaves if the seeds cannot survive the Jotnar digestive tract.
Do note that if the Jotnar develop technology that compensates for traits that lack, they will have much less pressure to evolve those traits. If they live in a savanna where height would help them see over the grass, they might instead burn and trample the grass around their settlements. If the tasty tree leaves become too high to reach, they might start building stepladders.
Alternatively, the Jotnar's height might be a result of founder effects. If a small population of unusually tall individuals became isolated from the main human population, their descendants will also, on average, be taller than the average human. If this happens repeatedly — for instance, if taller people regularly strike off on their own to found new settlements — then the effect will be magnified.
Similarly, population bottlenecks can greatly accelerate natural selection. In short, if there's some cataclysm that only the tall people survive (maybe a flood or tsunami where tall people are better able to climb to safety), the population that they rebuild will be taller than the one that came before.
Another possibility is that the Jotunn are simply healthier than anyone else on the planet. Humans have gradually become taller over the past centuries because we've created a better food supply. People used to run low on food over the winter, which stunted their growth. You say the Jotnar are technologically and magically advanced; if the rest of the world has a medieval food supply, they will likely be the tallest people on the planet.