High Recycling Value
Ensure that your rock and clay like substances from which all buildings are constructed have a high recycling value during the first few centuries after the destruction event. The event itself becomes less important, it could be just an "ordinarily massive"but I suggest a massive flood or firefrom a busted dam, made of this same material.
For example, your substance could be a rich metal ore which requires some moderate amount of work to dig up from the ground. The original inhabitants didn't know or care about metals, and just made buildings out of the raw ore (cut into bricks, or finely ground and mixed with some organic binding substance).
Place your city near a mountain pass or other naturally occurring "road". The availability of this building material was key to settling of this city, and supported its growth through the construction of dams to stabilize the water supply. The area is not otherwise rich in resources. Bursting of the largest dam could be part of the destruction event.
At the next major regional war, years or decades later, the ruins alongside the road become an easy target for recycling of the valuable material. It's a godsend for the army that takes it: the ore has already been dug up from the ground and cut into neat little blocks, ready to be melted down!
All the easily-available valuable material is depleted before the end of the war. That will prevent formation of a new settlement in the area, as the armies move on to their next strategic objectives.
Similarly, all smaller artifacts such as cooking pots and utensils, should be made of this same material which begs to be melted down and turned into weapons.
Travelers and treasure hunters will continue to extract whatever they find in the area, but will not resettle, due to the lack of a reliable water supply and insufficient materials to build a new dam. The exact location will be forgotten over time but the legends will live on and treasure hunters will continue to search along the road. Evidence from the area's use as a road and army supply route will further obscure the true history of what happened.
In much less than 4,000 years, all evidence of what was previously there or what happened, is gone.