Wild variety
Unfortunately I can give you only a part of the answer. And the answer is that it's heavily dependent on the climate.
It is safe to say mundane buildings wouldn't last long. Wood or mud or simple stone houses would last maybe a decade or so. People would build with anything they get their hands on and not go for the perfect long lasting wood, except if it happens to be available closeby. In addition, the craftsmanship leaves a lot to be desired. The majority would have little insulation and without people be very open to the elements and nature. This would degrade them quickly, leaving only some hints of what stood there.
Slightly higher tier buildings might be taverns, nobleman houses or simply many in certain city quarters. The ones made from wood would still only last a decade or two. With long lasting wood it might be a decade or two. If stone is used, it could go up to a few hundred years. The foundations are more likely to last longer, but after a thousand or a few thousand years there is likely nothing left but some scattered stones if they were used.
Fortifications would last the most. They are build to last and have the advantages of thickness, though height can be difficult at times. In addition, they start with an advantage that nature is generally kept at bay closer to the walls. A few hundred years it might stand, and thousands of years you can still be inside substantial ruins.
Now that the basics are out of the way, the second important thing to notice is the area they build it and the climate. Besides the material used, the area determines a lot of other weaknesses. A castle might stand for a thousand years, but things like erosion, foundations of salt being washed away or a flood might destroy the foundations.
Barring that, the climate can determine a ten thousand year hovel or a thirty year castle.
All zones with nature, especially trees, are highly vulnerable. Where tree roots can to through asphalt in our modern day, they can get more easily get in near or between stone and wood to grow and completely upheaval the structure.
Besides the nature, wind, water and Temperature differences are the greatest enemy. They erode with their oxigen and the expansion and shrinking of material can crack the material and even turn things into dust, leaving more open to the elements. A combination of water and Temperature is even worse. Cracks form, water gets in, freezes and destroys the material. Asphalt again is a good example. Made to last erosion from thousands, if not millions of cars driving over it. The cracks for the water to make the road safer in general, but one winter and the whole thing can crack, causing asphalt to warp, making dents and hills all over. This is especially dangerous if there's water and it's around freezing temperatures. During the day it might melt just enough, just to freeze again in the night, cracking the next layer to prepare it for the next assault. These things don't happen overnight, but with decades of preparation it could find a foothold and just rip towers in half.
More examples could be given, but most important is this:
It requires material highly resistant, if not impervious to rot. It needs to be build in a sturdy place that is unlikely to change. So no cliffs or earthquake area's. Little wind or water. Temperature must barely change. Little nature in the vicinity. Or if it doesn't need to be found above ground you have other options. Landslides or under water, like Willk is suggesting, can preserve it much, much longer. Then millions of years might be possible.
Otherwise the chance of just having substantial ruins is unlikely after a couple of thousand years, let alone a hundred thousand. Look at the current castles that haven't been maintained. 500 years later and many have become ruins.
