Totally different perceptual system.
Mythical creatures clearly don't prey on humans because they have to; they do so because they can. They have not evolved with human beings, they often come from a completely different set of dimensions.
While humans are fond of saying, "The cowl does not make the monk", thus proving they can distinguish where the man (monk) ends and the dress (cowl) begins, demons don't think in the same way - they don't even perceive in the same way.
To begin approaching this mindset you might think of an algorithm to model the human from their dress: you start from the "outside" (definition based on physical properties and similar to that of a locally differentiable manifold) and you proceed along the local normals until some specific property indicates a discontinuity. Now, the compact space lying along those normals is the human being.
This is how we might program a robotic surgeon or robo-policeman - that has no "perception" in the human sense - or something like that to "perceive" a human body. And that's very much like how demons "work".
Reversing one's dress completely changes its topology; to a demon, according to the algorithm above, the "wearer" within the dress is now the whole universe, and it is definitely not a human being. For all intents and purposes, the human has just disappeared. This kind of "disappearance" is common in the higher dimensions of demon-space, and is often used to attack. So, instinctively, the demon shies away, since to its eyes the situation has abruptly taken many of the characteristics of a demon trap.
As to how a demon's senses might be so thoroughly fooled this way, this is a characteristic they share with humans. Humans do not "see" what is in front of them, but synthesize a lot of semi-independent cues to infer shape, distance and motion. Not only several drugs can interfere with that process, but even seeing specific things can mess with the human visual perceptual system (sixty seconds of staring at this will give you, for the few seconds it takes the brain VPS to reset, what an alcohol addiction gives permanently - the illusion of something crawling under the skin of your hands or behind the wallpaper). The reverse is also true, which explains why "circles" can, in the appropriate circumstances, keep demons out, or demon-inspired dwellings have angles that are "subtly wrong", like an Ames room.
(This is the reason why some demonic rites mandate that the participants be naked. It allows demons a more "natural" - to them - management of their sensory space. Makes them comfortable, you might say).