People are very adaptable, with short memories. In biology, there are a few major restructurings of the brain; here is a brief summary.
The brain starts out with an overabundance of very dense neural connections, way too many to provide efficient functioning. These are still being built up to about age 3. From the age of 3 to 11, these are being connected by learning experience. Beginning about then (and puberty), the brain starts pruning these connections (lots of them, brutally), and starts a new process of myelinization which increases the speed of signals. There are more stages in the early teens, late teens, and the brain is not fully matured until the ages of around 24 to 28 years.
For the purpose of your story, it is our brain reorganizations that prevent (most of) us from remembering more than a few images before the age of 3, and make our childhood memories spotty: The pruning processes have caused us to lose connection to most of our early memories. (one speculation is that the few memories we do retain from our early years may be related to some early learning experience or learning moment.)
In your world, a culture in which we know our child is going to be taken away should mitigate any attachments to our child. We can love them and care for them, but there is a manageable level of loss felt if we have been expecting the separation for a long time. People handle their children going off to college or getting married and expecting that to be the end of their cohabitation. One thing you might do is have a "graduation ceremony" for the occasion of separation; the birth parents and adoptive parents together, etc. Analogous to a new marriage between families.
I strongly recommend transferring children before the age of two. By three years old, neural pruning is about to begin, or may have already begun. The initial stage of building new neural connections should be allowed to continue in the new home and new environment, with new (very basic) things to learn (new sounds, accents, vocalizations, images).
For example, the reason the Chinese language is quite difficult for native English speakers is because it relies on tonal and relative frequency variations that change the meaning of words. In English, we have very few of these; mostly just the rising intonation that signifies a question. (for example, Joe says "It was purple." The two responses, "Purple?" and "Purple." are understood differently by English speakers strictly based on the frequency inflection.) The actual neural connections that native Chinese-speaking children preserve is because they are hearing these intonations in infancy with regularity, so connections are formed and this sensory apparatus is preserved: In native English speakers, it is not developed or preserved because it has no consistent utility, and as adults, it is very difficult to learn. (This has nothing to do with genetics: children from all over the world have grown up in China, adopted or because their non-Chinese parents live and work in China, and they also develop and preserve these neural structures in the "machinery" of their auditory senses.)
Before age two would be ideal, but chances are, if done in the first six years, the children will not remember their birth parents, birth home, or nearly anything about their first six years: Just like the majority of us, today. I doubt they would be traumatized, or their parents would be.
There might be a problem with people (women, obviously) having many children out of egoism, because they don't have to support them. So an interesting rule would be that if you have children, you must pay child support to the State for your children after adoption, and all adoptive parents receive such support from the State. So your payment is effectively canceled by adoption. That might create a problem of over-adoption for financial interest, but that could be mitigated by limiting the total payment for adopted children. Not necessarily the number that can be adopted.
Alternatively, you can institute a socialist layer for children, specifically: It is paid for through child support payments, the State provides a full day of child-care and/or education for all children, one that includes full medical care and full nutrition on site, including frequent inspections for any abuse and mistreatment. Such inspectors can be in-home for infants still being raised to adoptive age by their birth parents.
To be brutally honest, you will also have to come up with solutions for the less-adoptable disabled children (physically or mentally) and those with birth deformities. Such children (IRL) are extremely difficult to place in adoptive homes (many adoptive parents are seeking the perfect child they wish they could have, so any significant deviation from ideal, whether a deformity or not, means some other kid is chosen), and in a more primitive society might be viewed as something evil or repellent. (I hasten to say it is not, biology is a bitch, and life is full of bad luck, including accidents and diseases, that are neither judgment or punishment. But superstitious primitive societies are not necessarily so enlightened, especially if magic is real as in the OP description.)
Added: I realized that this last effect (adoptive parents being averse to adopting children with problems) would, in the OP's world, create a de facto eugenics regime. The OP says:
If no parents step forward to claim the child, it will usually be raised in a caste-run institution comparable to a boarding school.
and:
... nobody is forced to raise a child if they don't want to ...
Since I think parents will get used to the idea that they don't raise their own offspring, and very few will adopt children with developmental problems or even just odd appearance, the "boarding school" residents will become a de facto "unwanted" sub-caste within each caste. This will be eugenics; a selective pressure toward appearances over all other human values, such as intelligence, empathy, altruism, etc. You will develop a race of beautiful children, dumb as rocks.
While we certainly do have a bias IRL toward beautiful people, the difference here (IRL) is that people raise their own kids, ugly or not, disabled or not. Such kids are generally treated with care and love by most parents (likely 85% based on other studies of responsibility and care), They do not just have them and expect them to disappear in two years, like in the OP's caste society.
The caste-run boarding schools could be hell holes; half hospitals for many kids, and basically prisons for the children neurally-normal but discarded anyway as too odd, ugly or malformed to be adopted. That will create pervasive resentment, hatreds, and increased anti-social behavior: Criminals and killers, unloved and unwanted, brought up in a factory and dumped on the street at adulthood.
