I'll try to answer in terms of time frames, since you asked about that specifically.
(If you haven't seen it yet, this answer may be of help to you.)
It sounds like you're envisioning a scenario completely devoid of human influence post-abandonment -- no looters or squatters or whatnot? Left unchecked, first the natural environment will move back in very quickly. Within a few months, any nicely-pruned grounds will have long lost their kempt appearance, and in only a couple of growth seasons the foliage will be clearly overtaking the walls. Animal life will move in essentially immediately.
Here's a neat article illustrating this in a town that is still even occupied.
Structurally, the stonework will last centuries; medieval castles continuously visited by tourists today are still intact with minimal (if any) maintenance. The rate at which any wooden constructions will rot away depends on several factors, such as climate, insects, and wood-loving fungi. Wood can last a long time if weather isn't a factor (in the more extreme cases, there are wooden tombs thousands of years old still standing, not to mention the wooden items like furniture and decorations inside). A fully intact, well-maintained roof suddenly abandoned will likely last at least two decades or more before any large problems begin to manifest. Such a roof could potentially stay more-or-less structurally sound for at least a hundred years; the main beams themselves could last a couple hundred, perhaps more. None of this takes into account severe weather or fire, of course.
Anything left undisturbed inside dry rooms will stay in good shape until exposed to the elements (which will happen, eventually).