ASOIAF's The Wall had actual physics problems, which were handwaved with magic. IIRC, the rest of your examples are just engineering, materials and construction problems.
The main thing you're looking for is compression vs tension loads. Stone (or unreinforced concrete) can handle huge compression loads, and it doesn't take a lot of engineering to design big structures or skill to build them, just an enormous amount of manual labor. They can also last indefinitely, which helps justify all that labor.
However, stone does not handle tension loads well at all, and that's what you look for any time you have holes or voids in a structure. Good engineering (e.g. arches) can partially solve that. Wood is better at tension loads, and it's obviously also a lot lighter than stone, but at the cost of permanence and/or maintenance.
A common tactic was to use stone walls supporting wood floors and roofs, which enables quite tall structures. However, there is still an engineering limit because the walls of lower levels must get thicker to support the ones above, and there is no point in going past a certain height because the lowest level's walls get so thick that it approaches completely solid.
Just eyeballing it, many fantasy structures seem to be well above that limit. Worse, many appear to have stone floors and roofs, but we can be charitable and say they're just thin tile over a wood structure. Still, it's hard to imagine many of them getting even half built without collapsing if using medieval materials and methods.