i am not sure i get the detail correctcorrectly, though i also agree with @seallussus if you can retake it that mean you have superior force or technologically.
ignoring the cost and materialmaterials, my suggestion is to build multiple layers of walls with different height put the outward or the middle layer to be the most tallest to support each wall that was under attack, this including the watchtower height putting roof is optional it can work against you, but it can also be a good lure to make enemy concentrate there, and collapse it when you throw bigger rock there from another layer/section of the wall using your trebuchet or catapult (only add such siege engine on the highest wall layer but if they manage to capture it you practically cant retake it other than destroy the wall) so if enemy taken one section/layer of wall you still have high ground advantage, even better if you can add water on the top, like some kind of roman aqueduct, so you can flood that section if its unsalvageable.
also use simple plank or improvise bridge or if want more fancy and harder to take make 2 opposite drawbridge to connect the other walls for your troop to move on, so if enemy attack and your guards and ppl need to evacuate they can easily getting rid of the plank quickly to disconnect it, or the enemy will need to take extra attempt to take both drawbridge, even better if you add multiple gate in each of the wall path/road to hold or slow them down, but i suggest use bar gate instead so you still can bombard them with projectiles rather than give them a checkpoint to take cover, also make the wall gate or entrance for your guard a bit higher maybe one floor higher with wooden stair or improvise stair to pastpass through so when enemy attack you can destroy it quickly to make it require extra effort to be entered and give another highground advantage for your guard, and add moat inside and outside the wallwalls perimeter including the gap of each wallwalls, you can use the drawbridge entrance to let the caravan pastpass through and connect to another section of the wall,walls.
also i assume the portal is inside the town and on the middle section of the quarters/districts right? so they practically get surrounded by the concentric wall, that itself should give the enemy a hard time to defend themselves and protect their siege engine while move closer to the wall, i suggest to clear any building from the portal perimeter for easy/clearer bombardment.
here example image for different multi layer wall height (dont take that to literal)
from:https://www.securityskeptic.com/2011/01/choke-and-screen-firewall-the-modern-day-concentric-castle.html
also here another example
from:https://en.wikipedia.org/wiki/Walls_of_Constantinople
my concern is enemy will try to dig pastpass through or destroy the wall using the tunnel when your guard not notice quickly to prevent it, so i guess you need to pave the town with concrete or plenty of deep stone foundation to make it harder to dig through or fill some section/layers of the moat with water, connect it to river is optional (this include the improvise roman aqueduct defense), i personally against it since that may become weakpoint in this wall defense but that depend on the town design and the wall fortification.
since portal is include here, i doubt you can starve them since they can just teleport their supply using the portal, but only taken one section probably wont be profitable or helpful economically for the enemy outside of raiding or pillaging which is a likely problem than invasion/occupation, since you can just barricade or block trade with them to destroy the economy flow, and they need to deal the same problem of inside invasion/occupation and pillaging too.