Would the size of someone determine how warm or cold they are?

Let's assume we have three subjects: a giant (50 ft tall), a human (6 ft tall) and a fairy (6 inches tall).

Common sense would lead you to believe that the giant would fair better in the cold, as opposed to the human and fairy and vice versa, assuming the terrain is a desert.

However, is this true to life? Assuming we do away with a square-cube law and scale a humanoid up and down by 8, would this have any effect on how they deal with temperature? Would size let them fair better or worse in different situations, or is size a factor that has no jurisdiction to how a being deals with temperature?

The ratio of surface area (amount exposed to the environment) vs. volume (containing temperate sensitive bits like organs) drops rapidly as size increases, leading to the conclusion that small things lose/gain more heat from their environment.

Having said that, elephants and mice both survive temperature changes. Their physiology is adapted to their body size. I suspect if you magically enlarged a mouse to elephant size, it wouldn't survive long for a host of reasons, heat management being a big one. But if you breed increasingly large mice over several centuries, the final elephant-sized offspring would probably survive.

Other factor (and a wild card) determining how various sized organisms fare with temperature changes is metabolism. As far as something like a metal statue cooling off, it is mass vs surface area as prior answers note.

This assumes homogeneity: a bronze statue. If the layer interfacing with the outside is a poor conductor of heat then this will slow conduction of heat from the core to the outside air or water. Blubber is an insulator. Feathers and fur are insulators. I am pretty sure fairies wear flesh-toned neoprene wetsuits; also an insulator.

Metabolic activity is essentially burning carbon compounds inside the body. In cold conditions a warm blooded animal can increase the metabolic rate to produce more heat. Human (usually babies but it can persist to adulthood) have special fat called brown fat with loads of mitochondria: the purpose is to make heat.

Cold blooded animals like tuna have other tricks by which they use increased metabolism to produce heat. The problem for very small creatures like hummingbirds is limited body space makes it hard to carry so much metabolic fuel on board, especially as carbohydrate. I propose that to overcome this problem your fairies can subsist largely on vodka and butter.

Proportions will matter more than height. For example, a short bulky man will fare better than a tall thin one. It's all about mass vs surface area (see Allen's rule). Neanderthals were shorter but far more stocky and robust, after all. Things like hair, insulation, metabolism, and proportions will be the most influential factors.

Size also affects how much you need to eat. A population of giants may have trouble finding enough food, and the lack of food in a desert is a bigger limit than lack of water. A fairy might not be able to find food regularly enough, but there are plenty of arctic rodents after all.