In a conventional force on force scenario, even combat engineers will have a great deal of difficulty (saps and tunnels can be bombarded from above to collapse them, for example).
So don't fight a conventional battle, but rather go 4GW on the enemy:
Fourth-generation warfare (4GW) uses all available networks -- political, economic, social, and military -- to convince the enemy’s political decision makers that their strategic goals are either unachievable or too costly for the perceived benefit. It is an evolved form of insurgency. Still rooted in the fundamental precept that superior political will, when properly employed, can defeat greater economic and military power, 4GW makes use of society’s networks to carry on its fight. Unlike previous generations of warfare, it does not attempt to win by defeating the enemy’s military forces. Instead, via the networks, it directly attacks the minds of enemy decision makers to destroy the enemy’s political will. Fourth-generation wars are lengthy -- measured in decades rather than months or years.
So the attackers sweep down from the hills and find....nothing. The knights and men at arms have fled, leaving only a confused and frightened peasantry. Rumours abound, but when questioned, the peasants have no usable information, only a mass of contradictory stories, many of which sound like nonsense. Unsatisfied, but seeing no real threat, the attackers move into the castle.
Men start disappearing at night. Horses are stolen. Raiders come to steal cows and burn crops. Trade caravans are robbed just outside the edge of the territory. Small treasures disappear from the castle (or are sometimes found in entirely different locations from where they were stored). The peasants are in an uproar, and try to crowd into the castle at night for protection. Yet, no matter how rapidly the new lords move, or if they send forces out to piquet the area, they never seem to find anyone. It is as if the night raiders have inside information, or hundreds of eyes watching their every move.
The new lords may suspect the peasants may have something to do with this, or the former owners are hidden in the population somehow, but face a quandary: torturing or abusing the peasants for information will only turn them against you, but right now they are already mistrusting the new lords because there is no protection from the night raiders. It is lose/lose either way.
Eventually, the new lords will be holed up in the castle, as if they are under siege. The food supplies are running low, morale is shot and lack of water and disease start to take their toll. At this point, your forces can come out of hiding and triumphal reappear to "save" the peasants and finish the siege. After all, the new lords are trapped in the low ground.....