The presence of creatures that mimic and prey on humans like that would be a really big impetus for humans to not live in small, isolated villages and group together in larger settlements. Living in small, isolated villages is about the worst thing you can do to defend against an intelligent predator that pretends to be a human. More people around means its easier for a mimic to get spotted if it slips up, and the mimics don't have an easy alibi of coming from another small village. Small isolated villages would be easy for a mimic to tear apart in a few feedings, especially if these villages are less than forty people or so. Especially because a small settlement is liable to be about half children and elderly. A village like that could easily disappear and if human communities are that small and isolated no one would notice until months later.
A small village is also going to be less separated from the wilderness and have less options for mimic-repellent. Lets say...burning fires repel mimics. A small village might have a handful of fires going at any moment, a small city would have hundreds. Or silver knives kill mimics. A small village has one, and if you lose it you're dead. A small city has many spares. A small village has a larger perimeter-to-area ratio that makes it harder to defend relative to its population size. So the question isn't "what keeps the mimics out" but "why do humans live in such a vulnerable state in the first place". One possibility might be food, there isn't enough food or water for humans to reliably live in large settlements.
Indeed, the best place for a mimic to hide would be within one of the human villages to avoid suspicion, and then going and preying on people from other villages to avoid producing an alarm. Maybe having a job as something like a woodcutter where they can justify going off alone, even better if they can do it in pairs and each mimic can vouch for the other. Travellers coming to the village would also be prime targets, as they could disappear and no one would notice. Indeed, if you can get multiple mimics pretending to be a group that means the whole "if you're alone, you aren't a mimic" means nothing, because a small group could have all been eaten and replaced at once.
There are a bunch of monsters from folklore who do this exact thing. One common story in these cultures is of a person doing something as simple as walking out of a room for five minutes or going to the bathroom and getting killed and replaced. It's amazing to think just how little we are directly aware of people around us.
The question also exists of whether the human communities don't end up mobbing each other The Monsters Are Due On Maple Street-style due to thinking another village could be unintentionally harboring mimics or even outright replaced by mimics. People in those kinds of situations, where they're surrounded by a predator they have to worry about every minute of every day but can't do anything practical about it, tend to be very paranoid and short-fused and liable to make stupid, extreme decisions in the belief that they're fixing the problem.
Indeed, if the mimics are all hiding in the forest and the people can't do much about them, they may be pushed to such extremes that they decide to burn the forest down and to heck with the consequences, hoping to survive on what food they have stored up until they can plant again.